Natural language processing with Python : [analyzing text with the natural language toolkit

This book offers a highly accessible introduction to natural language processing, the field that supports a variety of language technologies, from predictive text and email filtering to automatic summarization and translation. With it, you'll learn how to write Python programs that work with large collections of unstructured text. You'll access richly annotated datasets using a comprehensive range of linguistic data structures, and you'll understand the main algorithms for analyzing the content and structure of written communication. Packed with examples and exercises, Natural Language Processing with Python will help you: Extract information from unstructured text, either to guess the topic or identify "named entities" Analyze linguistic structure in text, including parsing and semantic analysis Access popular linguistic databases, including WordNet and treebanks Integrate techniques drawn from fields as diverse as linguistics and artificial intelligence This book will help you gain practical skills in natural language processing using the Python programming language and the Natural Language Toolkit (NLTK) open source library. If you're interested in developing web applications, analyzing multilingual news sources, or documenting endangered languages — or if you're simply curious to have a programmer's perspective on how human language works — you'll find Natural Language Processing with Python both fascinating and immensely useful. Table of Contents......Page 3 Preface......Page 7 Emphasis......Page 8 Organization......Page 9 Why Python?......Page 10 Software Requirements......Page 11 Natural Language Toolkit (NLTK)......Page 12 For Instructors......Page 13 Using Code Examples......Page 15 How to Contact Us......Page 16 Royalties......Page 17 1.1 Computing with Language: Texts and Words......Page 19 Getting Started with Python......Page 20 Getting Started with NLTK......Page 21 Searching Text......Page 22 Counting Vocabulary......Page 25 Lists......Page 28 Indexing Lists......Page 30 Variables......Page 32 Strings......Page 33 1.3 Computing with Language: Simple Statistics......Page 34 Frequency Distributions......Page 35 Fine-Grained Selection of Words......Page 37 Collocations and Bigrams......Page 38 Counting Other Things......Page 39 Conditionals......Page 40 Operating on Every Element......Page 42 Nested Code Blocks......Page 43 Looping with Conditions......Page 44 1.5 Automatic Natural Language Understanding......Page 45 Pronoun Resolution......Page 46 Machine Translation......Page 47 Spoken Dialogue Systems......Page 49 Textual Entailment......Page 50 1.6 Summary......Page 51 1.7 Further Reading......Page 52 1.8 Exercises......Page 53 2.1 Accessing Text Corpora......Page 56 Gutenberg Corpus......Page 57 Brown Corpus......Page 59 Reuters Corpus......Page 61 Inaugural Address Corpus......Page 62 Annotated Text Corpora......Page 63 Corpora in Other Languages......Page 65 Text Corpus Structure......Page 66 Loading Your Own Corpus......Page 68 Counting Words by Genre......Page 69 Plotting and Tabulating Distributions......Page 70 Generating Random Text with Bigrams......Page 72 Creating Programs with a Text Editor......Page 73 Functions......Page 74 2.4 Lexical Resources......Page 76 Wordlist Corpora......Page 77 A Pronouncing Dictionary......Page 80 Comparative Wordlists......Page 82 Shoebox and Toolbox Lexicons......Page 83 Senses and Synonyms......Page 84 The WordNet Hierarchy......Page 86 More Lexical Relations......Page 87 Semantic Similarity......Page 88 2.7 Further Reading......Page 90 2.8 Exercises......Page 91 Chapter 3. Processing Raw Text......Page 95 Electronic Books......Page 96 Dealing with HTML......Page 97 Processing Search Engine Results......Page 98 Processing RSS Feeds......Page 99 Reading Local Files......Page 100 Capturing User Input......Page 101 The NLP Pipeline......Page 102 Basic Operations with Strings......Page 103 Accessing Individual Characters......Page 105 Accessing Substrings......Page 106 The Difference Between Lists and Strings......Page 108 3.3 Text Processing with Unicode......Page 109 Extracting Encoded Text from Files......Page 110 3.4 Regular Expressions for Detecting Word Patterns......Page 113 Using Basic Metacharacters......Page 114 Ranges and Closures......Page 115 Doing More with Word Pieces......Page 118 Finding Word Stems......Page 120 Searching Tokenized Text......Page 121 Stemmers......Page 123 Lemmatization......Page 124 Simple Approaches to Tokenization......Page 125 Further Issues with Tokenization......Page 127 Sentence Segmentation......Page 128 Word Segmentation......Page 129 From Lists to Strings......Page 132 Strings and Formats......Page 133 Lining Things Up......Page 134 Text Wrapping......Page 136 3.10 Summary......Page 137 3.11 Further Reading......Page 138 3.12 Exercises......Page 139 Chapter 4. Writing Structured Programs......Page 145 Assignment......Page 146 Equality......Page 148 4.2 Sequences......Page 149 Operating on Sequence Types......Page 150 Combining Different Sequence Types......Page 152 Generator Expressions......Page 153 Python Coding Style......Page 154 Procedural Versus Declarative Style......Page 155 Some Legitimate Uses for Counters......Page 157 4.4 Functions: The Foundation of Structured Programming......Page 158 Function Inputs and Outputs......Page 159 Parameter Passing......Page 160 Variable Scope......Page 161 Checking Parameter Types......Page 162 Functional Decomposition......Page 163 Documenting Functions......Page 164 Functions As Arguments......Page 165 Accumulative Functions......Page 166 Higher-Order Functions......Page 167 Named Arguments......Page 168 Structure of a Python Module......Page 170 Multimodule Programs......Page 171 Sources of Error......Page 172 Debugging Techniques......Page 174 Defensive Programming......Page 175 Recursion......Page 176 Space-Time Trade-offs......Page 179 Dynamic Programming......Page 181 4.8 A Sample of Python Libraries......Page 183 Matplotlib......Page 184 NetworkX......Page 185 csv......Page 186 NumPy......Page 187 4.9 Summary......Page 188 4.11 Exercises......Page 189 5.1 Using a Tagger......Page 194 Reading Tagged Corpora......Page 196 A Simplified Part-of-Speech Tagset......Page 198 Nouns......Page 199 Verbs......Page 200 Adjectives and Adverbs......Page 201 Exploring Tagged Corpora......Page 202 Indexing Lists Versus Dictionaries......Page 204 Dictionaries in Python......Page 205 Default Dictionaries......Page 208 Incrementally Updating a Dictionary......Page 209 Complex Keys and Values......Page 211 Inverting a Dictionary......Page 212 The Default Tagger......Page 213 The Regular Expression Tagger......Page 214 The Lookup Tagger......Page 215 Evaluation......Page 216 Unigram Tagging......Page 217 General N-Gram Tagging......Page 218 Combining Taggers......Page 220 Performance Limitations......Page 221 5.6 Transformation-Based Tagging......Page 223 5.7 How to Determine the Category of a Word......Page 225 New Words......Page 226 Morphology in Part-of-Speech Tagsets......Page 227 5.8 Summary......Page 228 5.9 Further Reading......Page 229 5.10 Exercises......Page 230 6.1 Supervised Classification......Page 235 Gender Identification......Page 236 Choosing the Right Features......Page 238 Document Classification......Page 241 Part-of-Speech Tagging......Page 243 Exploiting Context......Page 244 Sequence Classification......Page 245 Sentence Segmentation......Page 247 Recognizing Textual Entailment......Page 249 The Test Set......Page 251 Precision and Recall......Page 253 Confusion Matrices......Page 254 Cross-Validation......Page 255 6.4 Decision Trees......Page 256 Entropy and Information Gain......Page 257 6.5 Naive Bayes Classifiers......Page 259 Underlying Probabilistic Model......Page 261 Zero Counts and Smoothing......Page 262 The Naivete of Independence......Page 263 6.6 Maximum Entropy Classifiers......Page 264 The Maximum Entropy Model......Page 265 Maximizing Entropy......Page 266 6.7 Modeling Linguistic Patterns......Page 268 What Do Models Tell Us?......Page 269 6.9 Further Reading......Page 270 6.10 Exercises......Page 271 7.1 Information Extraction......Page 274 Information Extraction Architecture......Page 276 Noun Phrase Chunking......Page 277 Chunking with Regular Expressions......Page 279 Exploring Text Corpora......Page 280 Chinking......Page 281 Representing Chunks: Tags Versus Trees......Page 282 Reading IOB Format and the CoNLL-2000 Chunking Corpus......Page 283 Simple Evaluation and Baselines......Page 285 Training Classifier-Based Chunkers......Page 287 Building Nested Structure with Cascaded Chunkers......Page 290 Trees......Page 292 Tree Traversal......Page 293 7.5 Named Entity Recognition......Page 294 7.6 Relation Extraction......Page 297 7.7 Summary......Page 298 7.9 Exercises......Page 299 Chapter 8. Analyzing Sentence Structure......Page 303 Linguistic Data and Unlimited Possibilities......Page 304 Ubiquitous Ambiguity......Page 305 Beyond n-grams......Page 307 A Simple Grammar......Page 310 Writing Your Own Grammars......Page 312 Recursion in Syntactic Structure......Page 313 8.4 Parsing with Context-Free Grammar......Page 314 Recursive Descent Parsing......Page 315 Shift-Reduce Parsing......Page 316 The Left-Corner Parser......Page 318 Well-Formed Substring Tables......Page 319 8.5 Dependencies and Dependency Grammar......Page 322 Valency and the Lexicon......Page 324 Scaling Up......Page 326 Treebanks and Grammars......Page 327 Pernicious Ambiguity......Page 329 Weighted Grammar......Page 330 8.7 Summary......Page 333 8.9 Exercises......Page 334 9.1 Grammatical Features......Page 339 Syntactic Agreement......Page 341 Using Attributes and Constraints......Page 343 Terminology......Page 347 9.2 Processing Feature Structures......Page 349 Subsumption and Unification......Page 353 Subcategorization......Page 356 Heads Revisited......Page 359 Auxiliary Verbs and Inversion......Page 360 Unbounded Dependency Constructions......Page 361 Case and Gender in German......Page 365 9.4 Summary......Page 368 9.5 Further Reading......Page 369 9.6 Exercises......Page 370 Querying a Database......Page 373 Natural Language, Semantics, and Logic......Page 377 10.2 Propositional Logic......Page 380 Syntax......Page 384 First-Order Theorem Proving......Page 387 Summarizing the Language of First-Order Logic......Page 388 Truth in Model......Page 389 Individual Variables and Assignments......Page 390 Quantification......Page 392 Quantifier Scope Ambiguity......Page 393 Model Building......Page 395 Compositional Semantics in Feature-Based Grammar......Page 397 The λ-Calculus......Page 398 Quantified NPs......Page 402 Transitive Verbs......Page 403 Quantifier Ambiguity Revisited......Page 406 Discourse Representation Theory......Page 409 Discourse Processing......Page 412 10.6 Summary......Page 414 10.7 Further Reading......Page 415 10.8 Exercises......Page 416 The Structure of TIMIT......Page 419 Notable Design Features......Page 421 Fundamental Data Types......Page 423 Three Corpus Creation Scenarios......Page 424 Quality Control......Page 425 Curation Versus Evolution......Page 426 Obtaining Data from Word Processor Files......Page 428 Obtaining Data from Spreadsheets and Databases......Page 430 Converting Data Formats......Page 431 Deciding Which Layers of Annotation to Include......Page 432 Standards and Tools......Page 433 Special Considerations When Working with Endangered Languages......Page 434 Using XML for Linguistic Structures......Page 437 The Role of XML......Page 438 The ElementTree Interface......Page 439 Using ElementTree for Accessing Toolbox Data......Page 441 Formatting Entries......Page 442 Adding a Field to Each Entry......Page 443 Validating a Toolbox Lexicon......Page 444 OLAC: Open Language Archives Community......Page 447 11.8 Further Reading......Page 449 11.9 Exercises......Page 450 Afterword: The Language Challenge......Page 452 Language Processing Versus Symbol Processing......Page 453 Contemporary Philosophical Divides......Page 454 NLTK Roadmap......Page 455 Envoi.........Page 458 Bibliography......Page 459 NLTK Index......Page 469 General Index......Page 473 This book offers a highly accessible introduction to Natural Language Processing, the field that underpins a variety of language technologies, ranging from predictive text and email filtering to automatic summarization and translation. With "Natural Language Processing with Python", you'll learn how to write Python programs to work with large collections of unstructured text. You'll access richly-annotated datasets using a comprehensive range of linguistic data structures. And you'll understand the main algorithms for analyzing the content and structure of written communication. Packed with examples and exercises, "Natural Language Processing with Python" will help you: extract information from unstructured text, to guess the topic or identify 'named entities'; analyze linguistic structure in text, including parsing and semantic analysis; access popular linguistic databases, including WordNet and treebanks; and, integrate techniques drawn from fields as diverse as linguistics and artificial intelligence. Perfect for individual study, or as a classroom and workshop textbook, this book will help you gain practical skills in Natural Language Processing using the Python programming language and the Natural Language Toolkit (NLTK) open source library. If you're interested in developing Web applications, analyzing multilingual news sources, documenting endangered languages, or if you are simply curious to have a programmer's perspective on how human language works, you will find "Natural Language Processing with Python" both fascinating and immensely useful The series builds an extensive collection of high quality descriptions of languages around the world. Each volume offers a comprehensive grammatical description of a single language together with fully analyzed sample texts and, if appropriate, a word list and other relevant information which is available on the language in question. There are no restrictions as to language family or area, and although special attention is paid to hitherto undescribed languages, new and valuable treatments of better known languages are also included. No theoretical model is imposed on the authors; the only criterion is a high standard of scientific quality. To discuss your book idea or submit a proposal, please contact Birgit Sievert. This book offers a highly accessible introduction to Natural Language Processing, the field that underpins a variety of language technologies ranging from predictive text and email filtering to automatic summarization and translation. You'll learn how to write Python programs to analyze the structure and meaning of texts, drawing on techniques from the fields of linguistics and artificial intelligence. This Is An Introduction To Natural Language Processing, Which Supports A Variety Of Language Technologies, From Predictive Text And Email Filtering To Automatic Summarization And Translation. Steven Bird, Ewan Klein, And Edward Loper. Analyzing Text With The Natural Language Toolkit--cover. Includes Bibliographical References (p. 449-458) And Indexes.