Natural language processing with Python : [analyzing text with the natural language toolkit

This book offers a highly accessible introduction to natural language processing, the field that supports a variety of language technologies, from predictive text and email filtering to automatic summarization and translation. With it, you'll learn how to write Python programs that work with large collections of unstructured text. You'll access richly annotated datasets using a comprehensive range of linguistic data structures, and you'll understand the main algorithms for analyzing the content and structure of written communication. Packed with examples and exercises, Natural Language Processing with Python will help you: Extract information from unstructured text, either to guess the topic or identify "named entities" Analyze linguistic structure in text, including parsing and semantic analysis Access popular linguistic databases, including WordNet and treebanks Integrate techniques drawn from fields as diverse as linguistics and artificial intelligence This book will help you gain practical skills in natural language processing using the Python programming language and the Natural Language Toolkit (NLTK) open source library. If you're interested in developing web applications, analyzing multilingual news sources, or documenting endangered languages — or if you're simply curious to have a programmer's perspective on how human language works — you'll find Natural Language Processing with Python both fascinating and immensely useful. Table of Contents 7 Preface 11 Audience 12 Emphasis 12 What You Will Learn 13 Organization 13 Why Python? 14 Software Requirements 15 Natural Language Toolkit (NLTK) 16 For Instructors 17 Conventions Used in This Book 19 Using Code Examples 19 Safari庐 Books Online 20 How to Contact Us 20 Acknowledgments 21 Royalties 21 Chapter聽1.聽Language Processing and Python 23 1.1聽 Computing with Language: Texts and Words 23 Getting Started with Python 24 Getting Started with NLTK 25 Searching Text 26 Counting Vocabulary 29 1.2聽 A Closer Look at Python: Texts as Lists of Words 32 Lists 32 Indexing Lists 34 Variables 36 Strings 37 1.3聽 Computing with Language: Simple Statistics 38 Frequency Distributions 39 Fine-Grained Selection of Words 41 Collocations and Bigrams 42 Counting Other Things 43 1.4聽 Back to Python: Making Decisions and Taking Control 44 Conditionals 44 Operating on Every Element 46 Nested Code Blocks 47 Looping with Conditions 48 1.5聽 Automatic Natural Language Understanding 49 Word Sense Disambiguation 50 Pronoun Resolution 50 Generating Language Output 51 Machine Translation 51 Spoken Dialogue Systems 53 Textual Entailment 54 Limitations of NLP 55 1.6聽 Summary 55 1.7聽 Further Reading 56 1.8聽 Exercises 57 Chapter聽2.聽Accessing Text Corpora and Lexical Resources 61 2.1聽 Accessing Text Corpora 61 Gutenberg Corpus 62 Web and Chat Text 64 Brown Corpus 64 Reuters Corpus 66 Inaugural Address Corpus 67 Annotated Text Corpora 68 Corpora in Other Languages 70 Text Corpus Structure 71 Loading Your Own Corpus 73 2.2聽 Conditional Frequency Distributions 74 Conditions and Events 74 Counting Words by Genre 74 Plotting and Tabulating Distributions 75 Generating Random Text with Bigrams 77 2.3聽 More Python: Reusing Code 78 Creating Programs with a Text Editor 78 Functions 79 Modules 81 2.4聽 Lexical Resources 81 Wordlist Corpora 82 A Pronouncing Dictionary 85 Comparative Wordlists 87 Shoebox and Toolbox Lexicons 88 2.5聽 WordNet 89 Senses and Synonyms 89 The WordNet Hierarchy 91 More Lexical Relations 92 Semantic Similarity 93 2.6聽 Summary 95 2.7聽 Further Reading 95 2.8聽 Exercises 96 Chapter聽3.聽Processing Raw Text 101 3.1聽 Accessing Text from the Web and from Disk 102 Electronic Books 102 Dealing with HTML 103 Processing Search Engine Results 104 Processing RSS Feeds 105 Reading Local Files 106 Extracting Text from PDF, MSWord, and Other Binary Formats 107 Capturing User Input 107 The NLP Pipeline 108 3.2聽 Strings: Text Processing at the Lowest Level 109 Basic Operations with Strings 109 Printing Strings 111 Accessing Individual Characters 111 Accessing Substrings 112 More Operations on Strings 114 The Difference Between Lists and Strings 114 3.3聽 Text Processing with Unicode 115 What Is Unicode? 116 Extracting Encoded Text from Files 116 Using Your Local Encoding in Python 119 3.4聽 Regular Expressions for Detecting Word Patterns 119 Using Basic Metacharacters 120 Ranges and Closures 121 3.5聽 Useful Applications of Regular Expressions 124 Extracting Word Pieces 124 Doing More with Word Pieces 124 Finding Word Stems 126 Searching Tokenized Text 127 3.6聽 Normalizing Text 129 Stemmers 129 Lemmatization 130 3.7聽 Regular Expressions for Tokenizing Text 131 Simple Approaches to Tokenization 131 NLTK鈥檚 Regular Expression Tokenizer 133 Further Issues with Tokenization 133 3.8聽 Segmentation 134 Sentence Segmentation 134 Word Segmentation 135 3.9聽 Formatting: From Lists to Strings 138 From Lists to Strings 138 Strings and Formats 139 Lining Things Up 140 Writing Results to a File 142 Text Wrapping 142 3.10聽 Summary 143 3.11聽 Further Reading 144 3.12聽 Exercises 145 Chapter聽4.聽Writing Structured Programs 151 4.1聽 Back to the Basics 152 Assignment 152 Equality 154 Conditionals 155 4.2聽 Sequences 155 Operating on Sequence Types 156 Combining Different Sequence Types 158 Generator Expressions 159 4.3聽 Questions of Style 160 Python Coding Style 160 Procedural Versus Declarative Style 161 Some Legitimate Uses for Counters 163 4.4聽 Functions: The Foundation of Structured Programming 164 Function Inputs and Outputs 165 Parameter Passing 166 Variable Scope 167 Checking Parameter Types 168 Functional Decomposition 169 Documenting Functions 170 4.5聽 Doing More with Functions 171 Functions As Arguments 171 Accumulative Functions 172 Higher-Order Functions 173 Named Arguments 174 4.6聽 Program Development 176 Structure of a Python Module 176 Multimodule Programs 177 Sources of Error 178 Debugging Techniques 180 Defensive Programming 181 4.7聽 Algorithm Design 182 Recursion 182 Space-Time Trade-offs 185 Dynamic Programming 187 4.8聽 A Sample of Python Libraries 189 Matplotlib 190 NetworkX 191 csv 192 NumPy 193 Other Python Libraries 194 4.9聽 Summary 194 4.10聽 Further Reading 195 4.11聽 Exercises 195 Chapter聽5.聽Categorizing and Tagging Words 201 5.1聽 Using a Tagger 201 5.2聽 Tagged Corpora 203 Representing Tagged Tokens 203 Reading Tagged Corpora 203 A Simplified Part-of-Speech Tagset 205 Nouns 206 Verbs 207 Adjectives and Adverbs 208 Unsimplified Tags 209 Exploring Tagged Corpora 209 5.3聽 Mapping Words to Properties Using Python Dictionaries 211 Indexing Lists Versus Dictionaries 211 Dictionaries in Python 212 Defining Dictionaries 215 Default Dictionaries 215 Incrementally Updating a Dictionary 216 Complex Keys and Values 218 Inverting a Dictionary 219 5.4聽 Automatic Tagging 220 The Default Tagger 220 The Regular Expression Tagger 221 The Lookup Tagger 222 Evaluation 223 5.5聽 N-Gram Tagging 224 Unigram Tagging 224 Separating the Training and Testing Data 225 General N-Gram Tagging 225 Combining Taggers 227 Tagging Unknown Words 228 Storing Taggers 228 Performance Limitations 228 Tagging Across Sentence Boundaries 230 5.6聽 Transformation-Based Tagging 230 5.7聽 How to Determine the Category of a Word 232 Morphological Clues 233 Syntactic Clues 233 Semantic Clues 233 New Words 233 Morphology in Part-of-Speech Tagsets 234 5.8聽 Summary 235 5.9聽 Further Reading 236 5.10聽 Exercises 237 Chapter聽6.聽Learning to Classify Text 243 6.1聽 Supervised Classification 243 Gender Identification 244 Choosing the Right Features 246 Document Classification 249 Part-of-Speech Tagging 251 Exploiting Context 252 Sequence Classification 253 Other Methods for Sequence Classification 255 6.2聽 Further Examples of Supervised Classification 255 Sentence Segmentation 255 Identifying Dialogue Act Types 257 Recognizing Textual Entailment 257 Scaling Up to Large Datasets 259 6.3聽 Evaluation 259 The Test Set 259 Accuracy 261 Precision and Recall 261 Confusion Matrices 262 Cross-Validation 263 6.4聽 Decision Trees 264 Entropy and Information Gain 265 6.5聽 Naive Bayes Classifiers 267 Underlying Probabilistic Model 269 Zero Counts and Smoothing 270 Non-Binary Features 271 The Naivete of Independence 271 The Cause of Double-Counting 272 6.6聽 Maximum Entropy Classifiers 272 The Maximum Entropy Model 273 Maximizing Entropy 274 Generative Versus Conditional Classifiers 276 6.7聽 Modeling Linguistic Patterns 276 What Do Models Tell Us? 277 6.8聽 Summary 278 6.9聽 Further Reading 278 6.10聽 Exercises 279 Chapter聽7.聽Extracting Information from Text 283 7.1聽 Information Extraction 283 Information Extraction Architecture 285 7.2聽 Chunking 286 Noun Phrase Chunking 286 Tag Patterns 288 Chunking with Regular Expressions 288 Exploring Text Corpora 289 Chinking 290 Representing Chunks: Tags Versus Trees 291 7.3聽 Developing and Evaluating Chunkers 292 Reading IOB Format and the CoNLL-2000 Chunking Corpus 292 Simple Evaluation and Baselines 294 Training Classifier-Based Chunkers 296 7.4聽 Recursion in Linguistic Structure 299 Building Nested Structure with Cascaded Chunkers 299 Trees 301 Tree Traversal 302 7.5聽 Named Entity Recognition 303 7.6聽 Relation Extraction 306 7.7聽 Summary 307 7.8聽 Further Reading 308 7.9聽 Exercises 308 Chapter聽8.聽Analyzing Sentence Structure 313 8.1聽 Some Grammatical Dilemmas 314 Linguistic Data and Unlimited Possibilities 314 Ubiquitous Ambiguity 315 8.2聽 What鈥檚 the Use of Syntax? 317 Beyond n-grams 317 8.3聽 Context-Free Grammar 320 A Simple Grammar 320 Writing Your Own Grammars 322 Recursion in Syntactic Structure 323 8.4聽 Parsing with Context-Free Grammar 324 Recursive Descent Parsing 325 Shift-Reduce Parsing 326 The Left-Corner Parser 328 Well-Formed Substring Tables 329 8.5聽 Dependencies and Dependency Grammar 332 Valency and the Lexicon 334 Scaling Up 336 8.6聽 Grammar Development 337 Treebanks and Grammars 337 Pernicious Ambiguity 339 Weighted Grammar 340 8.7聽 Summary 343 8.8聽 Further Reading 344 8.9聽 Exercises 344 Chapter聽9.聽Building Feature-Based Grammars 349 9.1聽 Grammatical Features 349 Syntactic Agreement 351 Using Attributes and Constraints 353 Terminology 357 9.2聽 Processing Feature Structures 359 Subsumption and Unification 363 9.3聽 Extending a Feature-Based Grammar 366 Subcategorization 366 Heads Revisited 369 Auxiliary Verbs and Inversion 370 Unbounded Dependency Constructions 371 Case and Gender in German 375 9.4聽 Summary 378 9.5聽 Further Reading 379 9.6聽 Exercises 380 Chapter聽10.聽Analyzing the Meaning of Sentences 383 10.1聽 Natural Language Understanding 383 Querying a Database 383 Natural Language, Semantics, and Logic 387 10.2聽 Propositional Logic 390 10.3聽 First-Order Logic 394 Syntax 394 First-Order Theorem Proving 397 Summarizing the Language of First-Order Logic 398 Truth in Model 399 Individual Variables and Assignments 400 Quantification 402 Quantifier Scope Ambiguity 403 Model Building 405 10.4聽 The Semantics of English Sentences 407 Compositional Semantics in Feature-Based Grammar 407 The 位-Calculus 408 Quantified NPs 412 Transitive Verbs 413 Quantifier Ambiguity Revisited 416 10.5聽 Discourse Semantics 419 Discourse Representation Theory 419 Discourse Processing 422 10.6聽 Summary 424 10.7聽 Further Reading 425 10.8聽 Exercises 426 Chapter聽11.聽Managing Linguistic Data 429 11.1聽 Corpus Structure: A Case Study 429 The Structure of TIMIT 429 Notable Design Features 431 Fundamental Data Types 433 11.2聽 The Life Cycle of a Corpus 434 Three Corpus Creation Scenarios 434 Quality Control 435 Curation Versus Evolution 436 11.3聽 Acquiring Data 438 Obtaining Data from the Web 438 Obtaining Data from Word Processor Files 438 Obtaining Data from Spreadsheets and Databases 440 Converting Data Formats 441 Deciding Which Layers of Annotation to Include 442 Standards and Tools 443 Special Considerations When Working with Endangered Languages 444 11.4聽 Working with XML 447 Using XML for Linguistic Structures 447 The Role of XML 448 The ElementTree Interface 449 Using ElementTree for Accessing Toolbox Data 451 Formatting Entries 452 11.5聽 Working with Toolbox Data 453 Adding a Field to Each Entry 453 Validating a Toolbox Lexicon 454 11.6聽 Describing Language Resources Using OLAC Metadata 457 What Is Metadata? 457 OLAC: Open Language Archives Community 457 11.7聽 Summary 459 11.8聽 Further Reading 459 11.9聽 Exercises 460 Afterword: The Language Challenge 463 Language Processing Versus Symbol Processing 464 Contemporary Philosophical Divides 465 NLTK Roadmap 466 Envoi... 469 Bibliography 471 NLTK Index 481 General Index 485 This book offers a highly accessible introduction to Natural Language Processing, the field that underpins a variety of language technologies, ranging from predictive text and email filtering to automatic summarization and translation. With "Natural Language Processing with Python", you'll learn how to write Python programs to work with large collections of unstructured text. You'll access richly-annotated datasets using a comprehensive range of linguistic data structures. And you'll understand the main algorithms for analyzing the content and structure of written communication. Packed with examples and exercises, "Natural Language Processing with Python" will help you: extract information from unstructured text, to guess the topic or identify 'named entities'; analyze linguistic structure in text, including parsing and semantic analysis; access popular linguistic databases, including WordNet and treebanks; and, integrate techniques drawn from fields as diverse as linguistics and artificial intelligence. Perfect for individual study, or as a classroom and workshop textbook, this book will help you gain practical skills in Natural Language Processing using the Python programming language and the Natural Language Toolkit (NLTK) open source library. If you're interested in developing Web applications, analyzing multilingual news sources, documenting endangered languages, or if you are simply curious to have a programmer's perspective on how human language works, you will find "Natural Language Processing with Python" both fascinating and immensely useful The series builds an extensive collection of high quality descriptions of languages around the world. Each volume offers a comprehensive grammatical description of a single language together with fully analyzed sample texts and, if appropriate, a word list and other relevant information which is available on the language in question. There are no restrictions as to language family or area, and although special attention is paid to hitherto undescribed languages, new and valuable treatments of better known languages are also included. No theoretical model is imposed on the authors; the only criterion is a high standard of scientific quality. To discuss your book idea or submit a proposal, please contact Birgit Sievert. ;Natural Language Processing with Python КНИГИ ; ОС и БД Название: Natural Language Processing with Python Автор: Steven Bird, Ewan Klein, Edward Loper Издательство: O'Reilly ISBN: 0-596-51649-5 Год: 2009 Формат: PDF Размер: 3.66 MB Страниц: 502 Качество: Хорошее Язык: АнглийскийОписание книги:This book offers a highly accessible introduction to Natural Language Processing, the field that underpins a variety of language technologies ranging from predictive text and email filtering to automatic summarization and translation. You'll learn how to write Python programs to analyze the structure and meaning of texts, drawing on techniques from the fields of linguistics and artificial intelligence.Скачать c .com 85 Extremely outdated !This book offers a highly accessible introduction to Natural Language Processing, the field that underpins a variety of language technologies ranging from predictive text and email filtering to automatic summarization and translation. You'll learn how to write Python programs to analyze the structure and meaning of texts, drawing on techniques from the fields of linguistics and artificial intelligence.Скачать c .com 85 This Is An Introduction To Natural Language Processing, Which Supports A Variety Of Language Technologies, From Predictive Text And Email Filtering To Automatic Summarization And Translation. Steven Bird, Ewan Klein, And Edward Loper. Analyzing Text With The Natural Language Toolkit--cover. Includes Bibliographical References (p. 449-458) And Indexes.