Web Scraping with Python: Data Extraction from the Modern Web

If programming is magic, then web scraping is surely a form of wizardry. By writing a simple automated program, you can query web servers, request data, and parse it to extract the information you need. This thoroughly updated third edition not only introduces you to web scraping but also serves as a comprehensive guide to scraping almost every type of data from the modern web. Part I focuses on web scraping mechanics: using Python to request information from a web server, performing basic handling of the server's response, and interacting with sites in an automated fashion. Part II explores a variety of more specific tools and applications to fit any web scraping scenario you're likely to encounter. • Parse complicated HTML pages • Develop crawlers with the Scrapy framework • Learn methods to store the data you scrape • Read and extract data from documents • Clean and normalize badly formatted data • Read and write natural languages • Crawl through forms and logins • Scrape JavaScript and crawl through APIs • Use and write image-to-text software • Avoid scraping traps and bot blockers • Use scrapers to test your website Cover Copyright Table of Contents Preface What Is Web Scraping? Why Web Scraping? About This Book Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Part I. Building Scrapers Chapter 1. How the Internet Works Networking Physical Layer Data Link Layer Network Layer Transport Layer Session Layer Presentation Layer Application Layer HTML CSS JavaScript Watching Websites with Developer Tools Chapter 2. The Legalities and Ethics of Web Scraping Trademarks, Copyrights, Patents, Oh My! Copyright Law Trespass to Chattels The Computer Fraud and Abuse Act robots.txt and Terms of Service Three Web Scrapers eBay v. Bidder’s Edge and Trespass to Chattels United States v. Auernheimer and the Computer Fraud and Abuse Act Field v. Google: Copyright and robots.txt Chapter 3. Applications of Web Scraping Classifying Projects E-commerce Marketing Academic Research Product Building Travel Sales SERP Scraping Chapter 4. Writing Your First Web Scraper Installing and Using Jupyter Connecting An Introduction to BeautifulSoup Installing BeautifulSoup Running BeautifulSoup Connecting Reliably and Handling Exceptions Chapter 5. Advanced HTML Parsing Another Serving of BeautifulSoup find() and find_all() with BeautifulSoup Other BeautifulSoup Objects Navigating Trees Regular Expressions Regular Expressions and BeautifulSoup Accessing Attributes Lambda Expressions You Don’t Always Need a Hammer Chapter 6. Writing Web Crawlers Traversing a Single Domain Crawling an Entire Site Collecting Data Across an Entire Site Crawling Across the Internet Chapter 7. Web Crawling Models Planning and Defining Objects Dealing with Different Website Layouts Structuring Crawlers Crawling Sites Through Search Crawling Sites Through Links Crawling Multiple Page Types Thinking About Web Crawler Models Chapter 8. Scrapy Installing Scrapy Initializing a New Spider Writing a Simple Scraper Spidering with Rules Creating Items Outputting Items The Item Pipeline Logging with Scrapy More Resources Chapter 9. Storing Data Media Files Storing Data to CSV MySQL Installing MySQL Some Basic Commands Integrating with Python Database Techniques and Good Practice “Six Degrees” in MySQL Email Part II. Advanced Scraping Chapter 10. Reading Documents Document Encoding Text Text Encoding and the Global Internet CSV Reading CSV Files PDF Microsoft Word and .docx Chapter 11. Working with Dirty Data Cleaning Text Working with Normalized Text Cleaning Data with Pandas Cleaning Indexing, Sorting, and Filtering More About Pandas Chapter 12. Reading and Writing Natural Languages Summarizing Data Markov Models Six Degrees of Wikipedia: Conclusion Natural Language Toolkit Installation and Setup Statistical Analysis with NLTK Lexicographical Analysis with NLTK Additional Resources Chapter 13. Crawling Through Forms and Logins Python Requests Library Submitting a Basic Form Radio Buttons, Checkboxes, and Other Inputs Submitting Files and Images Handling Logins and Cookies HTTP Basic Access Authentication Other Form Problems Chapter 14. Scraping JavaScript A Brief Introduction to JavaScript Common JavaScript Libraries Ajax and Dynamic HTML Executing JavaScript in Python with Selenium Installing and Running Selenium Selenium Selectors Waiting to Load XPath Additional Selenium WebDrivers Handling Redirects A Final Note on JavaScript Chapter 15. Crawling Through APIs A Brief Introduction to APIs HTTP Methods and APIs More About API Responses Parsing JSON Undocumented APIs Finding Undocumented APIs Documenting Undocumented APIs Combining APIs with Other Data Sources More About APIs Chapter 16. Image Processing and Text Recognition Overview of Libraries Pillow Tesseract NumPy Processing Well-Formatted Text Adjusting Images Automatically Scraping Text from Images on Websites Reading CAPTCHAs and Training Tesseract Training Tesseract Retrieving CAPTCHAs and Submitting Solutions Chapter 17. Avoiding Scraping Traps A Note on Ethics Looking Like a Human Adjust Your Headers Handling Cookies with JavaScript TLS Fingerprinting Timing Is Everything Common Form Security Features Hidden Input Field Values Avoiding Honeypots The Human Checklist Chapter 18. Testing Your Website with Scrapers An Introduction to Testing What Are Unit Tests? Python unittest Testing Wikipedia Testing with Selenium Interacting with the Site Chapter 19. Web Scraping in Parallel Processes Versus Threads Multithreaded Crawling Race Conditions and Queues More Features of the Threading Module Multiple Processes Multiprocess Crawling Communicating Between Processes Multiprocess Crawling—Another Approach Chapter 20. Web Scraping Proxies Why Use Remote Servers? Avoiding IP Address Blocking Portability and Extensibility Tor PySocks Remote Hosting Running from a Website-Hosting Account Running from the Cloud Moving Forward Web Scraping Proxies ScrapingBee ScraperAPI Oxylabs Zyte Additional Resources Index Colophon