Web Scraping with Python : Collecting Data from the Modern Web

ReadHowYouWant publishes a wide variety of best selling books in Large and Super Large fonts in partnership with leading publishers. EasyRead books are available in 11pt and 13pt. type. EasyRead Large books are available in 16pt, 16pt Bold, and 18pt Bold type. EasyRead Super Large books are available in 20pt. Bold and 24pt. Bold Type. You choose the format that is right for you. This is Volume Volume 8 of 10-Volume Set. To purchase the complete set, you will need to order the other volumes to find them, search for the following 9781427017420, 9781427028686, 9781427028730, 9781427029355, 9781427029836, 9781427029874, 9781427030061, 9781427030207, 9781427030214, 9781427030221 "War and Peace" is a brilliant prose epic by Tolstoy. He presents the events of 1812 as a moral crusade, when the Russians won against the Napoleonic onslaught through their adherence to virtue. He captures the essence of life with all its challenges, hardships and joys. The in-depth portrayal of numerous characters from all spheres of life lends it unparalleled realism. A true classic! To find more titles in your format, Search in Books using EasyRead and the size of the font that makes reading easier and more enjoyable for you. Preface What Is Web Scraping? Why Web Scraping? About This Book Conventions Used in This Book Using Code Examples Safari® Books Online How to Contact Us Acknowledgments I. Building Scrapers 1. Your First Web Scraper Connecting An Introduction to BeautifulSoup Installing BeautifulSoup Running BeautifulSoup Connecting Reliably 2. Advanced HTML Parsing You Don’t Always Need a Hammer Another Serving of BeautifulSoup find() and findAll() with BeautifulSoup Other BeautifulSoup Objects Navigating Trees Dealing with children and other descendants Dealing with siblings Dealing with your parents Regular Expressions Regular Expressions and BeautifulSoup Accessing Attributes Lambda Expressions Beyond BeautifulSoup 3. Starting to Crawl Traversing a Single Domain Crawling an Entire Site Collecting Data Across an Entire Site Crawling Across the Internet Crawling with Scrapy 4. Using APIs How APIs Work Common Conventions Methods Authentication Responses API Calls Echo Nest A Few Examples Twitter Getting Started A Few Examples Google APIs Getting Started A Few Examples Parsing JSON Bringing It All Back Home More About APIs 5. Storing Data Media Files Storing Data to CSV MySQL Installing MySQL Some Basic Commands Integrating with Python Database Techniques and Good Practice “Six Degrees” in MySQL Email 6. Reading Documents Document Encoding Text Text Encoding and the Global Internet A brief overview of encoding types Encodings in action CSV Reading CSV Files PDF Microsoft Word and .docx II. Advanced Scraping 7. Cleaning Your Dirty Data Cleaning in Code Data Normalization Cleaning After the Fact OpenRefine Installation Using OpenRefine Filtering Cleaning 8. Reading and Writing Natural Languages Summarizing Data Markov Models Six Degrees of Wikipedia: Conclusion Natural Language Toolkit Installation and Setup Statistical Analysis with NLTK Lexicographical Analysis with NLTK Additional Resources 9. Crawling Through Forms and Logins Python Requests Library Submitting a Basic Form Radio Buttons, Checkboxes, and Other Inputs Submitting Files and Images Handling Logins and Cookies HTTP Basic Access Authentication Other Form Problems 10. Scraping JavaScript A Brief Introduction to JavaScript Common JavaScript Libraries jQuery Google Analytics Google Maps Ajax and Dynamic HTML Executing JavaScript in Python with Selenium Handling Redirects 11. Image Processing and Text Recognition Overview of Libraries Pillow Tesseract Installing Tesseract NumPy Processing Well-Formatted Text Scraping Text from Images on Websites Reading CAPTCHAs and Training Tesseract Training Tesseract Retrieving CAPTCHAs and Submitting Solutions 12. Avoiding Scraping Traps A Note on Ethics Looking Like a Human Adjust Your Headers Handling Cookies Timing Is Everything Common Form Security Features Hidden Input Field Values Avoiding Honeypots The Human Checklist 13. Testing Your Website with Scrapers An Introduction to Testing What Are Unit Tests? Python unittest Testing Wikipedia Testing with Selenium Interacting with the Site Drag and drop Taking screenshots Unittest or Selenium? 14. Scraping Remotely Why Use Remote Servers? Avoiding IP Address Blocking Portability and Extensibility Tor PySocks Remote Hosting Running from a Website Hosting Account Running from the Cloud Additional Resources Moving Forward A. Python at a Glance Installation and “Hello, World!” B. The Internet at a Glance C. The Legalities and Ethics of Web Scraping Trademarks, Copyrights, Patents, Oh My! Copyright Law Trespass to Chattels The Computer Fraud and Abuse Act robots.txt and Terms of Service Three Web Scrapers eBay versus Bidder’s Edge and Trespass to Chattels United States v. Auernheimer and The Computer Fraud and Abuse Act Field v. Google: Copyright and robots.txt Index Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once. Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice. Learn how to parse complicated HTML pages Traverse multiple pages and sites Get a general overview of APIs and how they work Learn several methods for storing the data you scrape Download, read, and extract data from documents Use tools and techniques to clean badly formatted data Read and write natural languages Crawl through forms and logins Understand how to scrape JavaScript Learn image processing and text recognition Printing History April 2018: Second Edition 2018-03-20: First Release 2018-11-21: Second Release