Python for Data Analysis: Agile Tools for Real-World Data

Python for Data Analysis is concerned with the nuts and bolts of manipulating, processing, cleaning, and crunching data in Python. It is also a practical, modern introduction to scientific computing in Python, tailored for data-intensive applications. This is a book about the parts of the Python language and libraries you’ll need to effectively solve a broad set of data analysis problems. This book is not an exposition on analytical methods using Python as the implementation language. Written by Wes McKinney, the main author of the pandas library, this hands-on book is packed with practical cases studies. It’s ideal for analysts new to Python and for Python programmers new to scientific computing. Use the IPython interactive shell as your primary development environment Learn basic and advanced NumPy (Numerical Python) features Get started with data analysis tools in the pandas library Use high-performance tools to load, clean, transform, merge, and reshape data Create scatter plots and static or interactive visualizations with matplotlib Apply the pandas groupby facility to slice, dice, and summarize datasets Measure data by points in time, whether it’s specific instances, fixed periods, or intervals Learn how to solve problems in web analytics, social sciences, finance, and economics, through detailed examples Table of Contents 5 Preface 13 Conventions Used in This Book 13 Using Code Examples 14 Safari® Books Online 14 How to Contact Us 15 Chapter 1. Preliminaries 17 What Is This Book About? 17 Why Python for Data Analysis? 18 Python as Glue 18 Solving the “Two-Language” Problem 18 Why Not Python? 19 Essential Python Libraries 19 NumPy 20 pandas 20 matplotlib 21 IPython 21 SciPy 22 Installation and Setup 22 Windows 23 Apple OS X 25 GNU/Linux 26 Python 2 and Python 3 27 Integrated Development Environments (IDEs) 27 Community and Conferences 28 Navigating This Book 28 Code Examples 29 Data for Examples 29 Import Conventions 29 Jargon 29 Acknowledgements 30 Chapter 2. Introductory Examples 33 1.usa.gov data from bit.ly 33 Counting Time Zones in Pure Python 35 Counting Time Zones with pandas 37 MovieLens 1M Data Set 42 Measuring rating disagreement 46 US Baby Names 1880-2010 48 Analyzing Naming Trends 52 Measuring the increase in naming diversity 53 The “Last letter” Revolution 56 Boy names that became girl names (and vice versa) 58 Conclusions and The Path Ahead 59 Chapter 3. IPython: An Interactive Computing and Development Environment 61 IPython Basics 62 Tab Completion 63 Introspection 64 The %run Command 65 Interrupting running code 66 Executing Code from the Clipboard 66 IPython interaction with editors and IDEs 68 Keyboard Shortcuts 68 Exceptions and Tracebacks 69 Magic Commands 70 Qt-based Rich GUI Console 71 Matplotlib Integration and Pylab Mode 72 Using the Command History 74 Searching and Reusing the Command History 74 Input and Output Variables 74 Logging the Input and Output 75 Interacting with the Operating System 76 Shell Commands and Aliases 76 Directory Bookmark System 78 Software Development Tools 78 Interactive Debugger 78 Other ways to make use of the debugger 81 Timing Code: %time and %timeit 83 Basic Profiling: %prun and %run -p 84 Profiling a Function Line-by-Line 86 IPython HTML Notebook 88 Tips for Productive Code Development Using IPython 88 Reloading Module Dependencies 90 Code Design Tips 90 Keep relevant objects and data alive 91 Flat is better than nested 91 Overcome a fear of longer files 91 Advanced IPython Features 92 Making Your Own Classes IPython-friendly 92 Profiles and Configuration 93 Credits 94 Chapter 4. NumPy Basics: Arrays and Vectorized Computation 95 The NumPy ndarray: A Multidimensional Array Object 96 Creating ndarrays 97 Data Types for ndarrays 99 Operations between Arrays and Scalars 101 Basic Indexing and Slicing 102 Indexing with slices 104 Boolean Indexing 105 Fancy Indexing 108 Transposing Arrays and Swapping Axes 109 Universal Functions: Fast Element-wise Array Functions 111 Data Processing Using Arrays 113 Expressing Conditional Logic as Array Operations 114 Mathematical and Statistical Methods 116 Methods for Boolean Arrays 117 Sorting 117 Unique and Other Set Logic 118 File Input and Output with Arrays 119 Storing Arrays on Disk in Binary Format 119 Saving and Loading Text Files 120 Linear Algebra 121 Random Number Generation 122 Example: Random Walks 124 Simulating Many Random Walks at Once 125 Chapter 5. Getting Started with pandas 127 Introduction to pandas Data Structures 128 Series 128 DataFrame 131 Index Objects 136 Essential Functionality 138 Reindexing 138 Dropping entries from an axis 141 Indexing, selection, and filtering 141 Arithmetic and data alignment 144 Arithmetic methods with fill values 145 Operations between DataFrame and Series 146 Function application and mapping 148 Sorting and ranking 149 Axis indexes with duplicate values 152 Summarizing and Computing Descriptive Statistics 153 Correlation and Covariance 155 Unique Values, Value Counts, and Membership 157 Handling Missing Data 158 Filtering Out Missing Data 159 Filling in Missing Data 161 Hierarchical Indexing 163 Reordering and Sorting Levels 165 Summary Statistics by Level 166 Using a DataFrame’s Columns 166 Other pandas Topics 167 Integer Indexing 167 Panel Data 168 Chapter 6. Data Loading, Storage, and File Formats 171 Reading and Writing Data in Text Format 171 Reading Text Files in Pieces 176 Writing Data Out to Text Format 178 Manually Working with Delimited Formats 179 JSON Data 181 XML and HTML: Web Scraping 182 Parsing XML with lxml.objectify 185 Binary Data Formats 187 Using HDF5 Format 187 Reading Microsoft Excel Files 188 Interacting with HTML and Web APIs 189 Interacting with Databases 190 Storing and Loading Data in MongoDB 192 Chapter 7. Data Wrangling: Clean, Transform, Merge, Reshape 193 Combining and Merging Data Sets 193 Database-style DataFrame Merges 194 Merging on Index 198 Concatenating Along an Axis 201 Combining Data with Overlap 204 Reshaping and Pivoting 205 Reshaping with Hierarchical Indexing 206 Pivoting “long” to “wide” Format 208 Data Transformation 210 Removing Duplicates 210 Transforming Data Using a Function or Mapping 211 Replacing Values 212 Renaming Axis Indexes 213 Discretization and Binning 215 Detecting and Filtering Outliers 217 Permutation and Random Sampling 218 Computing Indicator/Dummy Variables 219 String Manipulation 221 String Object Methods 222 Regular expressions 223 Vectorized string functions in pandas 226 Example: USDA Food Database 228 Chapter 8. Plotting and Visualization 235 A Brief matplotlib API Primer 235 Figures and Subplots 236 Adjusting the spacing around subplots 239 Colors, Markers, and Line Styles 240 Ticks, Labels, and Legends 241 Setting the title, axis labels, ticks, and ticklabels 242 Adding legends 244 Annotations and Drawing on a Subplot 244 Saving Plots to File 247 matplotlib Configuration 247 Plotting Functions in pandas 248 Line Plots 248 Bar Plots 251 Histograms and Density Plots 254 Scatter Plots 255 Plotting Maps: Visualizing Haiti Earthquake Crisis Data 257 Python Visualization Tool Ecosystem 263 Chaco 264 mayavi 264 Other Packages 264 The Future of Visualization Tools? 265 Chapter 9. Data Aggregation and Group Operations 267 GroupBy Mechanics 268 Iterating Over Groups 271 Selecting a Column or Subset of Columns 272 Grouping with Dicts and Series 273 Grouping with Functions 274 Grouping by Index Levels 275 Data Aggregation 275 Column-wise and Multiple Function Application 278 Returning Aggregated Data in “unindexed” Form 280 Group-wise Operations and Transformations 280 Apply: General split-apply-combine 282 Suppressing the group keys 284 Quantile and Bucket Analysis 284 Example: Filling Missing Values with Group-specific Values 286 Example: Random Sampling and Permutation 287 Example: Group Weighted Average and Correlation 289 Example: Group-wise Linear Regression 290 Pivot Tables and Cross-Tabulation 291 Cross-Tabulations: Crosstab 293 Example: 2012 Federal Election Commission Database 294 Donation Statistics by Occupation and Employer 296 Bucketing Donation Amounts 299 Donation Statistics by State 301 Chapter 10. Time Series 305 Date and Time Data Types and Tools 306 Converting between string and datetime 307 Time Series Basics 309 Indexing, Selection, Subsetting 310 Time Series with Duplicate Indices 312 Date Ranges, Frequencies, and Shifting 313 Generating Date Ranges 314 Frequencies and Date Offsets 315 Week of month dates 317 Shifting (Leading and Lagging) Data 317 Shifting dates with offsets 318 Time Zone Handling 319 Localization and Conversion 320 Operations with Time Zone−aware Timestamp Objects 321 Operations between Different Time Zones 322 Periods and Period Arithmetic 323 Period Frequency Conversion 324 Quarterly Period Frequencies 325 Converting Timestamps to Periods (and Back) 327 Creating a PeriodIndex from Arrays 328 Resampling and Frequency Conversion 328 Downsampling 330 Open-High-Low-Close (OHLC) resampling 332 Resampling with GroupBy 332 Upsampling and Interpolation 332 Resampling with Periods 334 Time Series Plotting 335 Moving Window Functions 336 Exponentially-weighted functions 340 Binary Moving Window Functions 340 User-Defined Moving Window Functions 342 Performance and Memory Usage Notes 343 Chapter 11. Financial and Economic Data Applications 345 Data Munging Topics 345 Time Series and Cross-Section Alignment 346 Operations with Time Series of Different Frequencies 348 Using periods instead of timestamps 349 Time of Day and “as of” Data Selection 350 Splicing Together Data Sources 352 Return Indexes and Cumulative Returns 354 Group Transforms and Analysis 356 Group Factor Exposures 358 Decile and Quartile Analysis 359 More Example Applications 361 Signal Frontier Analysis 361 Future Contract Rolling 363 Rolling Correlation and Linear Regression 366 Chapter 12. Advanced NumPy 369 ndarray Object Internals 369 NumPy dtype Hierarchy 370 Advanced Array Manipulation 371 Reshaping Arrays 371 C versus Fortran Order 372 Concatenating and Splitting Arrays 373 Stacking helpers: r_ and c_ 375 Repeating Elements: Tile and Repeat 376 Fancy Indexing Equivalents: Take and Put 377 Broadcasting 378 Broadcasting Over Other Axes 380 Setting Array Values by Broadcasting 383 Advanced ufunc Usage 383 ufunc Instance Methods 384 Custom ufuncs 386 Structured and Record Arrays 386 Nested dtypes and Multidimensional Fields 387 Why Use Structured Arrays? 388 Structured Array Manipulations: numpy.lib.recfunctions 388 More About Sorting 389 Indirect Sorts: argsort and lexsort 390 Alternate Sort Algorithms 391 numpy.searchsorted: Finding elements in a Sorted Array 392 NumPy Matrix Class 393 Advanced Array Input and Output 395 Memory-mapped Files 395 HDF5 and Other Array Storage Options 396 Performance Tips 396 The Importance of Contiguous Memory 397 Other Speed Options: Cython, f2py, C 398 Appendix. Python Language Essentials 401 The Python Interpreter 402 The Basics 403 Language Semantics 403 Indentation, not braces 403 Everything is an object 404 Comments 404 Function and object method calls 405 Variables and pass-by-reference 405 Dynamic references, strong types 406 Attributes and methods 407 “Duck” typing 408 Imports 408 Binary operators and comparisons 409 Strictness versus laziness 410 Mutable and immutable objects 410 Scalar Types 411 Numeric types 411 Strings 412 Booleans 414 Type casting 415 None 415 Dates and times 415 Control Flow 416 if, elif, and else 416 for loops 417 while loops 418 pass 418 Exception handling 418 range and xrange 420 Ternary Expressions 421 Data Structures and Sequences 421 Tuple 422 Unpacking tuples 423 Tuple methods 423 List 424 Adding and removing elements 424 Concatenating and combining lists 425 Sorting 425 Binary search and maintaining a sorted list 426 Slicing 426 Built-in Sequence Functions 427 enumerate 428 sorted 428 zip 428 reversed 429 Dict 429 Creating dicts from sequences 431 Default values 431 Valid dict key types 432 Set 432 List, Set, and Dict Comprehensions 434 Nested list comprehensions 435 Functions 436 Namespaces, Scope, and Local Functions 436 Returning Multiple Values 438 Functions Are Objects 438 Anonymous (lambda) Functions 440 Closures: Functions that Return Functions 441 Extended Call Syntax with *args, **kwargs 442 Currying: Partial Argument Application 443 Generators 443 Generator expresssions 445 itertools module 445 Files and the operating system 446 Index 449 Python for Data Analysis is concerned with the nuts and bolts of manipulating, processing, cleaning, and crunching data in Python. It is also a practical, modern introduction to scientific computing in Python, tailored for data-intensive applications. This is a book about the parts of the Python language and libraries you’ll need to effectively solve a broad set of data analysis problems. This book is not an exposition on analytical methods using Python as the implementation language.Written by Wes McKinney, the main author of the pandas library, this hands-on book is packed with practical cases studies. It’s ideal for analysts new to Python and for Python programmers new to scientific computing.Use the IPython interactive shell as your primary development environmentLearn basic and advanced NumPy (Numerical Python) featuresGet started with data analysis tools in the pandas libraryUse high-performance tools to load, clean, transform, merge, and reshape dataCreate scatter plots and static or interactive visualizations with matplotlibApply the pandas groupby facility to slice, dice, and summarize datasetsMeasure data by points in time, whether it’s specific instances, fixed periods, or intervalsLearn how to solve problems in web analytics, social sciences, finance, and economics, through detailed examples __Python for Data Analysis__ is concerned with the nuts and bolts of manipulating, processing, cleaning, and crunching data in Python. It is also a practical, modern introduction to scientific computing in Python, tailored for data-intensive applications. This is a book about the parts of the Python language and libraries you’ll need to effectively solve a broad set of data analysis problems. This book is not an exposition on analytical methods using Python as the implementation language. Written by Wes McKinney, the main author of the pandas library, this hands-on book is packed with practical cases studies. It’s ideal for analysts new to Python and for Python programmers new to scientific computing. * Use the IPython interactive shell as your primary development environment * Learn basic and advanced NumPy (Numerical Python) features * Get started with data analysis tools in the pandas library * Use high-performance tools to load, clean, transform, merge, and reshape data * Create scatter plots and static or interactive visualizations with matplotlib * Apply the pandas groupby facility to slice, dice, and summarize datasets * Measure data by points in time, whether it’s specific instances, fixed periods, or intervals * Learn how to solve problems in web analytics, social sciences, finance, and economics, through detailed examples Finding great data analysts is difficult. Despite the explosive growth of data in industries ranging from manufacturing and retail to high technology, finance, and healthcare, learning and accessing data analysis tools has remained a challenge. This pragmatic guide will help train you in one of the most important tools in the field - Python. Filled with practical case studies, Python for Data Analysis demonstrates the nuts and bolts of manipulating, processing, cleaning, and crunching data with Python. It also serves as a modern introduction to scientific computing in Python for data-intensive applications. Learn about the growing field of data analysis from an expert in the community. Learn everything you need to start doing real data analysis work with Python Get the most complete instruction on the basics of the "modern scientific Python platform" Learn from an insider who builds tools for the scientific stack Get an excellent introduction for novices and a wealth of advanced methods for experienced analysis