Practical Web Scraping for Data Science : Best Practices and Examples with Python

This book provides a complete and modern guide to web scraping, using Python as the programming language, without glossing over important details or best practices. Written with a data science audience in mind, the book explores both scraping and the larger context of web technologies in which it operates, to ensure full understanding. The authors recommend web scraping as a powerful tool for any data scientist’s arsenal, as many data science projects start by obtaining an appropriate data set. Starting with a brief overview on scraping and real-life use cases, the authors explore the core concepts of HTTP, HTML, and CSS to provide a solid foundation. Along with a quick Python primer, they cover Selenium for JavaScript-heavy sites, and web crawling in detail. The book finishes with a recap of best practices and a collection of examples that bring together everything you've learned and illustrate various data science use cases.**What You'll Learn** * Leverage well-established best practices and commonly-used Python packages * Handle today's web, including JavaScript, cookies, and common web scraping mitigation techniques * Understand the managerial and legal concerns regarding web scraping **Who This Book is For** A data science oriented audience that is probably already familiar with Python or another programming language or analytical toolkit (R, SAS, SPSS, etc). Students or instructors in university courses may also benefit. Readers unfamiliar with Python will appreciate a quick Python primer in chapter 1 to catch up with the basics and provide pointers to other guides as well. Table of Contents 5 About the Authors 8 About the Technical Reviewer 9 Introduction 10 Part I: Web Scraping Basics 14 Chapter 1: Introduction 15 1.1 What Is Web Scraping? 15 1.1.1 Why Web Scraping for Data Science? 16 1.1.2 Who Is Using Web Scraping? 17 1.2 Getting Ready 20 1.2.1 Setting Up 20 1.2.2 A Quick Python Primer 21 Chapter 2: The Web Speaks HTTP 36 2.1 The Magic of Networking 36 2.2 The HyperText Transfer Protocol: HTTP 39 2.3 HTTP in Python: The Requests Library 45 2.4 Query Strings: URLs with Parameters 50 Chapter 3: Stirring the HTML and CSS Soup 60 3.1 Hypertext Markup Language: HTML 60 3.2 Using Your Browser as a Development Tool 62 3.3 Cascading Style Sheets: CSS 67 3.4 The Beautiful Soup Library 72 3.5 More on Beautiful Soup 83 Part II: Advanced Web Scraping 89 Chapter 4: Delving Deeper in HTTP 90 4.1 Working with Forms and POST Requests 90 4.2 Other HTTP Request Methods 106 4.3 More on Headers 109 4.4 Dealing with Cookies 117 4.5 Using Sessions with Requests 128 4.6 Binary, JSON, and Other Forms of Content 130 Chapter 5: Dealing with JavaScript 136 5.1 What Is JavaScript? 136 5.2 Scraping JavaScript 137 5.3 Scraping with Selenium 143 5.4 More on Selenium 157 Chapter 6: From Web Scraping to Web Crawling 164 6.1 What Is Web Crawling? 164 6.2 Web Crawling in Python 167 6.3 Storing Results in a Database 170 Part III: Managerial Concerns and Best Practices 182 Chapter 7: Managerial and Legal Concerns 183 7.1 The Data Science Process 183 7.2 Where Does Web Scraping Fit In? 187 7.3 Legal Concerns 189 Chapter 8: Closing Topics 195 8.1 Other Tools 195 8.1.1 Alternative Python Libraries 195 8.1.2 Scrapy 196 8.1.3 Caching 196 8.1.4 Proxy Servers 197 8.1.5 Scraping in Other Programming Languages 198 8.1.6 Command-Line Tools 199 8.1.7 Graphical Scraping Tools 199 8.2 Best Practices and Tips 201 Chapter 9: Examples 204 9.1 Scraping Hacker News 206 9.2 Using the Hacker News API 208 9.3 Quotes to Scrape 209 9.4 Books to Scrape 213 9.5 Scraping GitHub Stars 216 9.6 Scraping Mortgage Rates 221 9.7 Scraping and Visualizing IMDB Ratings 227 9.8 Scraping IATA Airline Information 229 9.9 Scraping and Analyzing Web Forum Interactions 235 9.10 Collecting and Clustering a Fashion Data Set 244 9.11 Sentiment Analysis of Scraped Amazon Reviews 248 9.12 Scraping and Analyzing News Articles 259 9.13 Scraping and Analyzing a Wikipedia Graph 278 9.14 Scraping and Visualizing a Board Members Graph 285 9.15 Breaking CAPTCHA’s Using Deep Learning 288 Index 306 This book provides a complete and modern guide to web scraping, using Python as the programming language, without glossing over important details or best practices. Written with a data science audience in mind, the book explores both scraping and the larger context of web technologies in which it operates, to ensure full understanding. The authors recommend web scraping as a powerful tool for any data scientist’s arsenal, as many data science projects start by obtaining an appropriate data set. Starting with a brief overview on scraping and real-life use cases, the authors explore the core concepts of HTTP, HTML, and CSS to provide a solid foundation. Along with a quick Python primer, they cover Selenium for JavaScript-heavy sites, and web crawling in detail. The book finishes with a recap of best practices and a collection of examples that bring together everything you've learned and illustrate various data science use cases. What You'll Learn Leverage well-established best practices and commonly-used Python packages Handle today's web, including JavaScript, cookies, and common web scraping mitigation techniques Understand the managerial and legal concerns regarding web scraping Who This Book is For A data science oriented audience that is probably already familiar with Python or another programming language or analytical toolkit (R, SAS, SPSS, etc). Students or instructors in university courses may also benefit. Readers unfamiliar with Python will appreciate a quick Python primer in chapter 1 to catch up with the basics and provide pointers to other guides as well. This book provides a complete and modern guide to web scraping, using Python as the programming language, without glossing over important details or best practices. Written with a data science audience in mind, the book explores both scraping and the larger context of web technologies in which it operates, to ensure full understanding. The authors recommend web scraping as a powerful tool for any data scientist's arsenal, as many data science projects start by obtaining an appropriate data set. Starting with a brief overview on scraping and real-life use cases, the authors explore the core concepts of HTTP, HTML, and CSS to provide a solid foundation. Along with a quick Python primer, they cover requests and Beautiful Soup, Selenium for JavaScript-heavy sites, and web crawling in detail. The book finishes with a recap of best practices and a collection of examples that bring together everything you've learned and illustrate various data science use cases Résumé : This book provides a complete and modern guide to web scraping, using Python as the programming language, without glossing over important details or best practices. Written with a data science audience in mind, the book explores both scraping and the larger context of web technologies in which it operates, to ensure full understanding. The authors recommend web scraping as a powerful tool for any data scientist's arsenal, as many data science projects start by obtaining an appropriate data set