Learning pandas - Python Data Discovery and Analysis Made Easy

Key Features Employ the use of pandas for data analysis closely to focus more on analysis and less on programming Get programmers comfortable in performing data exploration and analysis on Python using pandas Step-by-step demonstration of using Python and pandas with interactive and incremental examples to facilitate learning Book Description This learner's guide will help you understand how to use the features of pandas for interactive data manipulation and analysis. This book is your ideal guide to learning about pandas, all the way from installing it to creating one- and two-dimensional indexed data structures, indexing and slicing-and-dicing that data to derive results, loading data from local and Internet-based resources, and finally creating effective visualizations to form quick insights. You start with an overview of pandas and NumPy and then dive into the details of pandas, covering pandas' Series and DataFrame objects, before ending with a quick review of using pandas for several problems in finance. With the knowledge you gain from this book, you will be able to quickly begin your journey into the exciting world of data science and analysis. What You Will Learn Install pandas on Windows, Mac, and Linux using the Anaconda Python distribution Learn how pandas builds on NumPy to implement flexible indexed data Adopt pandas' Series and DataFrame objects to represent one- and two-dimensional data constructs Index, slice, and transform data to derive meaning from information Load data from files, databases, and web services Manipulate dates, times, and time series data Group, aggregate, and summarize data Visualize techniques for pandas and statistical data About the Author Michael Heydt is an independent consultant, educator, and trainer with nearly 30 years of professional software development experience, during which time, he focused on Agile software design and implementation using advanced technologies in multiple verticals, including media, finance, energy, and healthcare. Since 2005, he has specialized in building energy and financial trading systems for major investment banks on Wall Street and for several global energy-trading companies, utilizing .NET, C#, WPF, TPL, DataFlow, Python, R, Mono, iOS, and Android. His current interests include creating seamless applications using desktop, mobile, and wearable technologies, which utilize high-concurrency, high-availability, and real-time data analytics; augmented and virtual reality; cloud services; messaging; computer vision; natural user interfaces; and software-defined networks. He is the author of numerous technology articles, papers, and books. He is a frequent speaker at .NET user groups and various mobile and cloud conferences, and he regularly delivers webinars and conducts training courses on emerging and advanced technologies. Table of Content A Tour of pandas Installing pandas Numpy for pandas The pandas Series Object The pandas Dataframe Object Accessing Data Tidying up Your Data Combining and Reshaping Data Grouping and Aggregating Data Time-series Data Visualization Applications to Finance Cover 1 Copyright 3 Credits 4 About the Author 5 About the Reviewers 6 www.PacktPub.com 8 Table of Contents 10 Preface 16 Chapter 1: A Tour of pandas 22 pandas and why it is important 23 pandas and IPython Notebooks 24 Referencing pandas in the application 26 Primary pandas objects 27 The pandas Series object 27 The pandas DataFrame object 32 Loading data from files and the Web 39 Loading CSV data from files 39 Loading data from the Web 43 Simplicity of visualization of pandas data 45 Summary 46 Chapter 2: Installing pandas 48 Getting Anaconda 49 Installing Anaconda 50 Installing Anaconda on Linux 51 Installing Anaconda on Mac OS X 53 Installing Anaconda on Windows 54 Ensuring pandas is up to date 56 Running a small pandas sample in IPython 59 Starting the IPython Notebook server 60 Installing and running IPython Notebooks 64 Using Wakari for pandas 66 Summary 68 Chapter 3: NumPy for pandas 70 Installing and importing NumPy 71 Benefits and characteristics of NumPy arrays 71 Creating NumPy arrays and performing basic array operations 73 Selecting array elements 79 Logical operations on arrays 80 Slicing arrays 82 Reshaping arrays 86 Combining arrays 91 Splitting arrays 94 Useful numerical methods of NumPy arrays 100 Summary 103 Chapter 4: The pandas Series Object 104 The Series object 105 Importing pandas 106 Creating Series 106 Size, shape, uniqueness, and counts of values 112 Peeking at data with heads, tails, and take 114 Looking up values in Series 116 Alignment via index labels 121 Arithmetic operations 123 The special case of Not-A-Number (NaN) 127 Boolean selection 128 Reindexing a Series 131 Modifying a Series in-place 138 Slicing a Series 140 Summary 147 Chapter 5: The pandas DataFrame Object 148 Creating DataFrame from scratch 149 Example data 153 S&P 500 154 Monthly stock historical prices 156 Selecting columns of a DataFrame 157 Selecting rows and values of a DataFrame using the index 163 Slicing using the [] operator 163 Selecting rows by index label and location: .loc[] and .iloc[] 164 Selecting rows by index label and/or location: .ix[] 166 Scalar lookup by label or location using .at[] and .iat[] 167 Selecting rows of a DataFrame by Boolean selection 167 Modifying the structure and content of DataFrame 169 Renaming columns 169 Adding and inserting columns 171 Replacing the contents of a column 174 Deleting columns in a DataFrame 175 Adding rows to a DataFrame 178 Appending rows with .append() 178 Concatenating DataFrame objects with pd.concat() 180 Adding rows (and columns) via setting with enlargement 186 Removing rows from a DataFrame 187 Removing rows using .drop() 187 Removing rows using Boolean selection 189 Removing rows using a slice 190 Changing scalar values in a DataFrame 191 Arithmetic on a DataFrame 193 Resetting and reindexing 197 Hierarchical indexing 200 Summarized data and descriptive statistics 206 Summary 212 Chapter 6: Accessing Data 214 Setting up the IPython notebook 215 CSV and Text/Tabular format 215 The sample CSV data set 215 Reading a CSV file into a DataFrame 216 Specifying the index column when reading a CSV file 217 Data type inference and specification 217 Specifying column names 218 Specifying specific columns to load 219 Saving DataFrame to a CSV file 220 General field-delimited data 221 Handling noise rows in field-delimited data 222 Reading and writing data in an Excel format 225 Reading and writing JSON files 228 Reading HTML data from the Web 229 Reading and writing HDF5 format files 232 Accessing data on the web and in the cloud 234 Reading and writing from/to SQL databases 235 Reading data from remote data services 238 Reading stock data from Yahoo! and Google Finance 239 Retrieving data from Yahoo! Finance Options 240 Reading economic data from the Federal Reserve Bank of St. Louis 243 Accessing Kenneth French's data 246 Reading from the World Bank 247 Summary 255 Chapter 7: Tidying Up Your Data 256 What is tidying your data? 257 Setting up the IPython notebook 258 Working with missing data 258 Determining NaN values in Series and DataFrame objects 260 Selecting out or dropping missing data 262 How pandas handles NaN values in mathematical operations 267 Filling in missing data 269 Forward and backward filling of missing values 271 Filling using index labels 272 Interpolation of missing values 273 Handling duplicate data 276 Transforming Data 280 Mapping 280 Replacing values 282 Applying functions to transform data 285 Summary 290 Chapter 8: Combining and Reshaping Data 292 Setting up the IPython notebook 293 Concatenating data 293 Merging and joining data 302 An overview of merges 302 Specifying the join semantics of a merge operation 307 Pivoting 310 Stacking and unstacking 311 Stacking using nonhierarchical indexes 312 Unstacking using hierarchical indexes 314 Melting 319 Performance benefits of stacked data 320 Summary 321 Chapter 9: Grouping and Aggregating Data 324 Setting up the IPython notebook 325 The split, apply, and combine (SAC) pattern 325 Split 327 Data for the examples 327 Grouping by a single column's values 328 Accessing the results of grouping 329 Grouping using index levels 335 Apply 339 Applying aggregation functions to groups 339 The transformation of group data 343 An overview of transformation 343 Practical examples of transformation 347 Filtering groups 354 Discretization and Binning 356 Summary 362 Chapter 10: Time-series Data 364 Setting up the IPython notebook 365 Representation of dates, time, and intervals 366 The datetime, day, and time objects 366 Timestamp objects 368 Timedelta 370 Introducing time-series data 371 DatetimeIndex 371 Creating time-series data with specific frequencies 378 Calculating new dates using offsets 380 Date offsets 381 Anchored offsets 385 Representing durations of time using Period objects 387 The Period object 387 PeriodIndex 389 Handling holidays using calendars 393 Normalizing timestamps using time zones 394 Manipulating time-series data 400 Shifting and lagging 400 Frequency conversion 404 Up and down resampling 407 Time-series moving-window operations 412 Summary 416 Chapter 11: Visualization 418 Setting up the IPython notebook 419 Plotting basics with pandas 420 Creating time-series charts with .plot() 421 Adorning and styling your time-series plot 425 Adding a title and changing axes labels 425 Specifying the legend content and position 427 Specifying line colors, styles, thickness, and markers 429 Specifying tick mark locations and tick labels 433 Formatting axes tick date labels using formatters 436 Common plots used in statistical analyses 442 Bar plots 443 Histograms 445 Box and whisker charts 448 Area plots 449 Scatter plots 451 Density plot 453 The scatter plot matrix 454 Heatmaps 456 Multiple plots in a single chart 457 Summary 461 Chapter 12: Applications to Finance 462 Setting up the IPython notebook 463 Obtaining and organizing stock data from Yahoo! 463 Plotting time-series prices 468 Plotting volume-series data 470 Calculating the simple daily percentage change 472 Calculating simple daily cumulative returns 474 Resampling data from daily to monthly returns 476 Analyzing distribution of returns 478 Performing a moving-average calculation 481 The comparison of average daily returns across stocks 482 The correlation of stocks based on the daily percentage change of the closing price 486 Volatility calculation 488 Determining risk relative to expected returns 489 Summary 492 Index 494 Get to grips with pandas--a versatile and high-performance Python library for data manipulation, analysis, and discoveryAbout This Book* Get comfortable using pandas and Python as an effective data exploration and analysis tool* Explore pandas through a framework of data analysis, with an explanation of how pandas is well suited for the various stages in a data analysis process* A comprehensive guide to pandas with many of clear and practical examples to help you get up and using pandasWho This Book Is ForThis book is ideal for data scientists, data analysts, Python programmers who want to plunge into data analysis using pandas, and anyone with a curiosity about analyzing data. Some knowledge of statistics and programming will be helpful to get the most out of this book but not strictly required. Prior exposure to pandas is also not required. What You Will Learn* Understand how data analysts and scientists think about of the processes of gathering and understanding data* Learn how pandas can be used to support the end-to-end process of data analysis* Use pandas Series and DataFrame objects to represent single and multivariate data* Slicing and dicing data with pandas, as well as combining, grouping, and aggregating data from multiple sources* How to access data from external sources such as files, databases, and web services* Represent and manipulate time-series data and the many of the intricacies involved with this type of data* How to visualize statistical information* How to use pandas to solve several common data representation and analysis problems within financeIn DetailYou will learn how to use pandas to perform data analysis in Python. You will start with an overview of data analysis and iteratively progress from modeling data, to accessing data from remote sources, performing numeric and statistical analysis, through indexing and performing aggregate analysis, and finally to visualizing statistical data and applying pandas to finance. With the knowledge you gain from this book, you will quickly learn pandas and how it can empower you in the exciting world of data manipulation, analysis and science. Style and approach* Step-by-step instruction on using pandas within an end-to-end framework of performing data analysis* Practical demonstration of using Python and pandas using interactive and incremental examples

About This Book

• Learn to manipulate, visualize, and analyze a wide range of financial data with the help of built-in functions and programming in R
• Understand the concepts of financial engineering and create trading strategies for complex financial instruments
• Explore R for asset and liability management and capital adequacy modeling

Who This Book Is For

This book is intended for those who want to learn how to use R's capabilities to build models in quantitative finance at a more advanced level. If you wish to perfectly take up the rhythm of the chapters, you need to be at an intermediate level in quantitative finance and you also need to have a reasonable knowledge of R.

About This BookEmploy the use of pandas for data analysis closely to focus more on analysis and less on programmingGet programmers comfortable in performing data exploration and analysis on Python using pandasStep-by-step demonstration of using Python and pandas with interactive and incremental examples to facilitate learningWho This Book Is ForIf you are a Python programmer who wants to get started with performing data analysis using pandas and Python, this is the book for you. Some experience with statistical analysis would be helpful but is not mandatory. "Pandas is a popular Python package used for practical, real-world data analysis. It provides efficient, fast, high-performance data structures that make data exploration and analysis very easy. This learner's guide will help you through a comprehensive set of features provided by the pandas library to perform efficient data manipulation and analysis. You will learn how to use pandas to perform data anaylsis in Python."--Page 4 of cover