Web Scraping with Python : Collecting Data From the Modern Web

ReadHowYouWant publishes a wide variety of best selling books in Large and Super Large fonts in partnership with leading publishers. EasyRead books are available in 11pt and 13pt. type. EasyRead Large books are available in 16pt, 16pt Bold, and 18pt Bold type. EasyRead Super Large books are available in 20pt. Bold and 24pt. Bold Type. You choose the format that is right for you. This is Volume Volume 8 of 10-Volume Set. To purchase the complete set, you will need to order the other volumes to find them, search for the following 9781427017420, 9781427028686, 9781427028730, 9781427029355, 9781427029836, 9781427029874, 9781427030061, 9781427030207, 9781427030214, 9781427030221 "War and Peace" is a brilliant prose epic by Tolstoy. He presents the events of 1812 as a moral crusade, when the Russians won against the Napoleonic onslaught through their adherence to virtue. He captures the essence of life with all its challenges, hardships and joys. The in-depth portrayal of numerous characters from all spheres of life lends it unparalleled realism. A true classic! To find more titles in your format, Search in Books using EasyRead and the size of the font that makes reading easier and more enjoyable for you. Cover 1 Copyright 4 Table of Contents 5 Preface 9 What Is Web Scraping? 10 Why Web Scraping? 10 About This Book 11 Conventions Used in This Book 12 Using Code Examples 13 Safari® Books Online 14 How to Contact Us 14 Acknowledgments 15 Part I. Building Scrapers 17 Chapter 1. Your First Web Scraper 19 Connecting 19 An Introduction to BeautifulSoup 22 Installing BeautifulSoup 22 Running BeautifulSoup 24 Connecting Reliably 25 Chapter 2. Advanced HTML Parsing 29 You Don’t Always Need a Hammer 29 Another Serving of BeautifulSoup 30 find() and findAll() with BeautifulSoup 32 Other BeautifulSoup Objects 34 Navigating Trees 34 Regular Expressions 38 Regular Expressions and BeautifulSoup 43 Accessing Attributes 44 Lambda Expressions 44 Beyond BeautifulSoup 45 Chapter 3. Starting to Crawl 47 Traversing a Single Domain 47 Crawling an Entire Site 51 Collecting Data Across an Entire Site 54 Crawling Across the Internet 56 Crawling with Scrapy 61 Chapter 4. Using APIs 65 How APIs Work 66 Common Conventions 66 Methods 67 Authentication 68 Responses 68 API Calls 69 Echo Nest 70 A Few Examples 70 Twitter 71 Getting Started 72 A Few Examples 73 Google APIs 76 Getting Started 76 A Few Examples 77 Parsing JSON 79 Bringing It All Back Home 80 More About APIs 84 Chapter 5. Storing Data 87 Media Files 87 Storing Data to CSV 90 MySQL 92 Installing MySQL 93 Some Basic Commands 95 Integrating with Python 98 Database Techniques and Good Practice 101 “Six Degrees” in MySQL 103 Email 106 Chapter 6. Reading Documents 109 Document Encoding 109 Text 110 Text Encoding and the Global Internet 110 CSV 114 Reading CSV Files 114 PDF 116 Microsoft Word and .docx 118 Part II. Advanced Scraping 123 Chapter 7. Cleaning Your Dirty Data 125 Cleaning in Code 125 Data Normalization 128 Cleaning After the Fact 129 OpenRefine 130 Chapter 8. Reading and Writing Natural Languages 135 Summarizing Data 136 Markov Models 139 Six Degrees of Wikipedia: Conclusion 142 Natural Language Toolkit 145 Installation and Setup 145 Statistical Analysis with NLTK 146 Lexicographical Analysis with NLTK 148 Additional Resources 152 Chapter 9. Crawling Through Forms and Logins 153 Python Requests Library 153 Submitting a Basic Form 154 Radio Buttons, Checkboxes, and Other Inputs 156 Submitting Files and Images 157 Handling Logins and Cookies 158 HTTP Basic Access Authentication 160 Other Form Problems 160 Chapter 10. Scraping JavaScript 163 A Brief Introduction to JavaScript 164 Common JavaScript Libraries 165 Ajax and Dynamic HTML 167 Executing JavaScript in Python with Selenium 168 Handling Redirects 174 Chapter 11. Image Processing and Text Recognition 177 Overview of Libraries 178 Pillow 178 Tesseract 179 NumPy 180 Processing Well-Formatted Text 180 Scraping Text from Images on Websites 182 Reading CAPTCHAs and Training Tesseract 185 Training Tesseract 187 Retrieving CAPTCHAs and Submitting Solutions 190 Chapter 12. Avoiding Scraping Traps 193 A Note on Ethics 193 Looking Like a Human 194 Adjust Your Headers 195 Handling Cookies 197 Timing Is Everything 198 Common Form Security Features 199 Hidden Input Field Values 199 Avoiding Honeypots 200 The Human Checklist 202 Chapter 13. Testing Your Website with Scrapers 205 An Introduction to Testing 205 What Are Unit Tests? 206 Python unittest 206 Testing Wikipedia 207 Testing with Selenium 209 Interacting with the Site 210 Unittest or Selenium? 213 Chapter 14. Scraping Remotely 215 Why Use Remote Servers? 215 Avoiding IP Address Blocking 215 Portability and Extensibility 216 Tor 217 PySocks 218 Remote Hosting 219 Running from a Website Hosting Account 219 Running from the Cloud 220 Additional Resources 222 Moving Forward 222 Appendix A. Python at a Glance 225 Installation and “Hello, World!” 225 Appendix B. The Internet at a Glance 229 Appendix C. The Legalities and Ethics of Web Scraping 233 Trademarks, Copyrights, Patents, Oh My! 233 Copyright Law 234 Trespass to Chattels 235 The Computer Fraud and Abuse Act 237 robots.txt and Terms of Service 238 Three Web Scrapers 242 eBay versus Bidder’s Edge and Trespass to Chattels 242 United States v. Auernheimer and The Computer Fraud and Abuse Act 243 Field v. Google: Copyright and robots.txt 245 Index 247 About the Author 255 Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once. Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice. Learn how to parse complicated HTML pages Traverse multiple pages and sites Get a general overview of APIs and how they work Learn several methods for storing the data you scrape Download, read, and extract data from documents Use tools and techniques to clean badly formatted data Read and write natural languages Crawl through forms and logins Understand how to scrape JavaScript Learn image processing and text recognition Printing History April 2018: Second Edition 2018-03-20: First Release 2018-11-21: Second Release