You’ve learned the basics of Python, but how do you take your skills to the next stage? Even if you know enough to be productive, there are a number of features that can take you to the next level in Python. Pro Python explores concepts and features normally left to experimentation, allowing you to be even more productive and creative. In addition to pure code concerns, Pro Python will develop your programming techniques and approaches, which will help make you a better Python programmer. Not only will this book help your code, it will also help you understand and interact with the many established Python communities, or even start your own. Take your Python knowledge and coding skills to the next level. Write clean, innovative code that will be respected among your peers. Make your code do more with introspection and metaprogramming. Design complete frameworks and libraries (two are included in the book!). What you’ll learn Write strong Python code that will be respected in the Python community. Understand the reasons behind big design decisions in Python. Write programs that can reconfigure themselves in Python. Disguise your code as different types of objects in Python. Inspect just about any object in Python. Prepare your code for international audiences. Ensure code quality with rigorous testing. Who this book is for This book is for intermediate to advanced Python programmers who are looking to understand how and why Python works the way it does and how they can take their code to the next level. Table of Contents Principles and Philosophy Advanced Basics Functions Classes Common Protocols Object Management Strings Documentation Testing Distribution Sheets: A CSV Framework Prelim 1 Contents at a Glance 5 Contents 6 About the Author 18 About the Technical Reviewer 19 Acknowledgments 20 Introduction 21 Who This Book Is For 21 What You’ll Need 21 Source Code 21 Principles and Philosophy 23 The Zen of Python 23 Beautiful Is Better Than Ugly 24 Explicit Is Better Than Implicit 24 Simple Is Better Than Complex 25 Complex Is Better Than Complicated 25 Flat Is Better Than Nested 26 Sparse Is Better Than Dense 27 Readability Counts 27 Special Cases Aren’t Special Enough to Break the Rules 28 Although Practicality Beats Purity 28 Errors Should Never Pass Silently 29 Unless Explicitly Silenced 30 In the Face of Ambiguity, Refuse the Temptation to Guess 31 There Should Be One—and Preferably Only One— Obvious Way to Do It 32 Although That Way May Not Be Obvious at First Unless You’re Dutch 32 Now Is Better Than Never 33 Although Never Is Often Better Than Right Now 33 If the Implementation is Hard to Explain, It’s a Bad Idea 33 If the Implementation is Easy to Explain, It May Be a Good Idea 33 Namespaces Are One Honking Great Idea— Let’s Do More of Those! 34 Don’t Repeat Yourself 34 Loose Coupling 35 The Samurai Principle 35 The Pareto Principle 36 The Robustness Principle 36 Backward Compatibility 37 The Road to Python 3.0 38 Taking It With You 39 Advanced Basics 41 General Concepts 41 Iteration 41 Caching 42 Transparency 43 Control Flow 43 Catching Exceptions 43 Exception Chains 46 When Everything Goes Right 48 Proceeding Regardless of Exceptions 49 Optimizing Loops 51 The with Statement 51 Conditional Expressions 53 Iteration 55 Sequence Unpacking 56 List Comprehensions 57 Generator Expressions 58 Set Comprehensions 59 Dictionary Comprehensions 59 Chaining Iterables Together 60 Zipping Iterables Together 60 Collections 61 Sets 61 Named Tuples 65 Ordered Dictionaries 66 Dictionaries with Defaults 66 Importing Code 67 Fallback Imports 67 Importing from the Future 68 Using __all__ to Customize Imports 69 Relative Imports 70 The __import__() function 71 The importlib module 73 Taking It With You 74 Functions 75 Arguments 75 Planning for Flexibility 76 Variable Positional Arguments 76 Variable Keyword Arguments 77 Combining Different Kinds of Arguments 78 Invoking Functions with Variable Arguments 81 Preloading Arguments 82 Introspection 83 Example: Identifying Argument Values 84 Example: A More Concise Version 86 Example: Validating Arguments 88 Decorators 89 Closures 91 Wrappers 93 Decorators with Arguments 94 Decorators with—or without—Arguments 96 Example: Memoization 97 Example: A Decorator to Create Decorators 99 Function Annotations 100 Example: Type Safety 101 Factoring Out the Boilerplate 108 Example: Type Coercion 110 Annotating with Decorators 112 Example: Type Safety as a Decorator 112 Generators 116 Lambdas 118 Introspection 119 Identifying Object Types 120 Modules and Packages 120 Docstrings 121 Taking It With You 123 Classes 125 Inheritance 125 Multiple Inheritance 127 Method Resolution Order (MRO) 128 Example: C3 Algorithm 131 Using super() to Pass Control to Other Classes 137 Introspection 139 How Classes Are Created 141 Creating Classes at Runtime 142 Metaclasses 143 Example: Plugin Framework 144 Controlling the Namespace 147 Attributes 148 Properties 149 Descriptors 151 Methods 153 Unbound Methods 153 Bound Methods 154 Class Methods 155 Static Methods 156 Assigning Functions to Classes and Instances 157 Magic Methods 157 Creating Instances 158 Example: Automatic Subclasses 159 Dealing with Attributes 160 String Representations 162 Taking It With You 164 Common Protocols 165 Basic Operations 165 Mathematical Operations 166 Bitwise Operations 170 Variations 172 Numbers 174 Sign Operations 176 Comparison Operations 176 Iterables 177 Example: Repeatable Generators 180 Sequences 181 Mappings 186 Callables 187 Context Managers 188 Taking It With You 190 Object Management 191 Namespace Dictionary 192 Example: Borg Pattern 192 Example: Self-caching properties 195 Garbage Collection 198 Reference Counting 199 Cyclical References 200 Weak References 202 Pickling 204 Copying 208 Shallow Copies 209 Deep Copies 210 Taking It With You 212 Strings 213 Bytes 213 Simple Conversion: chr() and ord() 214 Complex Conversion: The Struct Module 215 Text 217 Unicode 218 Encodings 218 Simple Substitution 220 Formatting 223 Looking Up Values Within Objects 224 Distinguishing Types of Strings 224 Standard Format Specification 225 Example: Plain Text Table of Contents 226 Custom Format Specification 227 Taking It With You 228 Documentation 229 Proper Naming 229 Comments 230 Docstrings 230 Describe What the Function Does 231 Explain the Arguments 231 Don’t Forget the Return Value 231 Include Any Expected Exceptions 232 Documentation Outside the Code 232 Installation and Configuration 232 Tutorials 232 Reference Documents 232 Documentation Utilities 233 Formatting 234 Links 235 Sphinx 236 Taking It With You 237 Testing 239 Test-Driven Development (TDD) 239 Doctests 240 Formatting Code 240 Representing Output 240 Integrating With Documentation 241 Running Tests 242 The unittest module 243 Setting Up 243 Writing Tests 244 Other Comparisons 248 Testing Strings and Other Sequence Content 248 Testing Exceptions 249 Testing Identity 251 Tearing Down 251 Providing a Custom Test Class 252 Changing Test Behavior 252 Taking It With You 253 Distribution 255 Licensing 255 GNU General Public License (GPL) 255 Affero General Public License (AGPL) 256 GNU Lesser General Public License (LGPL) 257 Berkeley Software Distribution (BSD) License 257 Other Licenses 258 Packaging 258 setup.py 259 MANIFEST.in 261 The sdist command 262 Distribution 263 Taking It With You 264 Sheets: A CSV Framework 265 Building a Declarative Framework 266 Introducing Declarative Programming 266 To Build or Not to Build? 267 Building the Framework 268 Managing Options 269 Defining Fields 271 Attaching a Field to a Class 272 Adding a Metaclass 274 Bringing It Together 277 Ordering Fields 278 DeclarativeMeta.__prepare__() 278 Column.__init__() 280 Column.__new__() 283 CounterMeta.__call__() 284 Choosing an Option 285 Building a Field Library 285 StringField 286 IntegerColumn 287 FloatColumn 287 DecimalColumn 287 DateColumn 288 Getting Back to CSV 292 Checking Arguments 293 Populating Values 295 The Reader 297 The Writer 301 Taking It With You 303 Style Guide for Python 305 Introduction 305 A Foolish Consistency is the Hobgoblin of Little Minds 305 Code Layout 306 Indentation 306 Tabs or Spaces? 306 Maximum Line Length 306 Blank Lines 306 Encodings (PEP 263) 307 Imports 307 Whitespace in Expressions and Statements 308 Pet Peeves 308 Other Recommendations 309 Comments 310 Block Comments 311 Inline Comments 311 Documentation Strings 311 Version Bookkeeping 312 Naming Conventions 312 Descriptive: Naming Styles 312 Prescriptive: Naming Conventions Names to Avoid 313 Package and Module Names 313 Class Names 314 Exception Names 314 Global Variable Names 314 Function Names 314 Function and Method Arguments 314 Method Names and Instance Variables 314 Constants 315 Designing for Inheritance 315 Programming Recommendations 316 Copyright 319 Voting Guidelines 321 Abstract 321 Rationale 321 Voting Scores 321 Copyright 322 The Zen of Python 323 Abstract 323 The Zen of Python 323 Easter Egg 323 Copyright 324 Docstring Conventions 325 Abstract 325 Rationale 325 Specification 325 What is a Docstring? 325 One-Line Docstrings 326 Multi-Line Docstrings 327 Handling Docstring Indentation 328 Copyright 329 Acknowledgments 329 Backwards Compatibility Policy 331 Abstract 331 Rationale 331 Backwards Compatibility Rules 331 Making Incompatible Changes 332 Copyright 333 Python 3000 335 Abstract 335 Naming 335 PEP Numbering 335 Timeline 335 Compatibility and Transition 336 Implementation Language 337 Meta-Contributions 337 Copyright 337 Python Language Moratorium 339 Abstract 339 Rationale 339 Details 340 Cannot Change 340 Case-by-Case Exemptions 340 Allowed to Change 340 Retroactive 341 Extensions 341 Copyright 341 Index 343 Symbols and Numerics 343 A 343 B 344 C 345 D 347 E 349 F 349 H 351 G 351 I 351 L 353 K 353 M 353 N 356 O 356 P 357 R 358 S 359 T 361 U 362 V 362 X 363 Y 363 Z 363 W 363 Prelim......Page 1 Contents at a Glance......Page 5 Contents......Page 6 About the Author......Page 18 About the Technical Reviewer......Page 19 Acknowledgments......Page 20 Source Code......Page 21 The Zen of Python......Page 23 Explicit Is Better Than Implicit......Page 24 Complex Is Better Than Complicated......Page 25 Flat Is Better Than Nested......Page 26 Readability Counts......Page 27 Although Practicality Beats Purity......Page 28 Errors Should Never Pass Silently......Page 29 Unless Explicitly Silenced......Page 30 In the Face of Ambiguity, Refuse the Temptation to Guess......Page 31 Although That Way May Not Be Obvious at First Unless You’re Dutch......Page 32 If the Implementation is Easy to Explain, It May Be a Good Idea......Page 33 Don’t Repeat Yourself......Page 34 The Samurai Principle......Page 35 The Robustness Principle......Page 36 Backward Compatibility......Page 37 The Road to Python 3.0......Page 38 Taking It With You......Page 39 Iteration......Page 41 Caching......Page 42 Catching Exceptions......Page 43 Exception Chains......Page 46 When Everything Goes Right......Page 48 Proceeding Regardless of Exceptions......Page 49 The with Statement......Page 51 Conditional Expressions......Page 53 Iteration......Page 55 Sequence Unpacking......Page 56 List Comprehensions......Page 57 Generator Expressions......Page 58 Dictionary Comprehensions......Page 59 Zipping Iterables Together......Page 60 Sets......Page 61 Named Tuples......Page 65 Dictionaries with Defaults......Page 66 Fallback Imports......Page 67 Importing from the Future......Page 68 Using __all__ to Customize Imports......Page 69 Relative Imports......Page 70 The __import__() function......Page 71 The importlib module......Page 73 Taking It With You......Page 74 Arguments......Page 75 Variable Positional Arguments......Page 76 Variable Keyword Arguments......Page 77 Combining Different Kinds of Arguments......Page 78 Invoking Functions with Variable Arguments......Page 81 Preloading Arguments......Page 82 Introspection......Page 83 Example: Identifying Argument Values......Page 84 Example: A More Concise Version......Page 86 Example: Validating Arguments......Page 88 Decorators......Page 89 Closures......Page 91 Wrappers......Page 93 Decorators with Arguments......Page 94 Decorators with—or without—Arguments......Page 96 Example: Memoization......Page 97 Example: A Decorator to Create Decorators......Page 99 Function Annotations......Page 100 Example: Type Safety......Page 101 Factoring Out the Boilerplate......Page 108 Example: Type Coercion......Page 110 Example: Type Safety as a Decorator......Page 112 Generators......Page 116 Lambdas......Page 118 Introspection......Page 119 Modules and Packages......Page 120 Docstrings......Page 121 Taking It With You......Page 123 Inheritance......Page 125 Multiple Inheritance......Page 127 Method Resolution Order (MRO)......Page 128 Example: C3 Algorithm......Page 131 Using super() to Pass Control to Other Classes......Page 137 Introspection......Page 139 How Classes Are Created......Page 141 Creating Classes at Runtime......Page 142 Metaclasses......Page 143 Example: Plugin Framework......Page 144 Controlling the Namespace......Page 147 Attributes......Page 148 Properties......Page 149 Descriptors......Page 151 Unbound Methods......Page 153 Bound Methods......Page 154 Class Methods......Page 155 Static Methods......Page 156 Magic Methods......Page 157 Creating Instances......Page 158 Example: Automatic Subclasses......Page 159 Dealing with Attributes......Page 160 String Representations......Page 162 Taking It With You......Page 164 Basic Operations......Page 165 Mathematical Operations......Page 166 Bitwise Operations......Page 170 Variations......Page 172 Numbers......Page 174 Comparison Operations......Page 176 Iterables......Page 177 Example: Repeatable Generators......Page 180 Sequences......Page 181 Mappings......Page 186 Callables......Page 187 Context Managers......Page 188 Taking It With You......Page 190 Object Management......Page 191 Example: Borg Pattern......Page 192 Example: Self-caching properties......Page 195 Garbage Collection......Page 198 Reference Counting......Page 199 Cyclical References......Page 200 Weak References......Page 202 Pickling......Page 204 Copying......Page 208 Shallow Copies......Page 209 Deep Copies......Page 210 Taking It With You......Page 212 Bytes......Page 213 Simple Conversion: chr() and ord()......Page 214 Complex Conversion: The Struct Module......Page 215 Text......Page 217 Encodings......Page 218 Simple Substitution......Page 220 Formatting......Page 223 Distinguishing Types of Strings......Page 224 Standard Format Specification......Page 225 Example: Plain Text Table of Contents......Page 226 Custom Format Specification......Page 227 Taking It With You......Page 228 Proper Naming......Page 229 Docstrings......Page 230 Don’t Forget the Return Value......Page 231 Reference Documents......Page 232 Documentation Utilities......Page 233 Formatting......Page 234 Links......Page 235 Sphinx......Page 236 Taking It With You......Page 237 Test-Driven Development (TDD)......Page 239 Representing Output......Page 240 Integrating With Documentation......Page 241 Running Tests......Page 242 Setting Up......Page 243 Writing Tests......Page 244 Testing Strings and Other Sequence Content......Page 248 Testing Exceptions......Page 249 Tearing Down......Page 251 Changing Test Behavior......Page 252 Taking It With You......Page 253 GNU General Public License (GPL)......Page 255 Affero General Public License (AGPL)......Page 256 Berkeley Software Distribution (BSD) License......Page 257 Packaging......Page 258 setup.py......Page 259 MANIFEST.in......Page 261 The sdist command......Page 262 Distribution......Page 263 Taking It With You......Page 264 Sheets: A CSV Framework......Page 265 Introducing Declarative Programming......Page 266 To Build or Not to Build?......Page 267 Building the Framework......Page 268 Managing Options......Page 269 Defining Fields......Page 271 Attaching a Field to a Class......Page 272 Adding a Metaclass......Page 274 Bringing It Together......Page 277 DeclarativeMeta.__prepare__()......Page 278 Column.__init__()......Page 280 Column.__new__()......Page 283 CounterMeta.__call__()......Page 284 Building a Field Library......Page 285 StringField......Page 286 DecimalColumn......Page 287 DateColumn......Page 288 Getting Back to CSV......Page 292 Checking Arguments......Page 293 Populating Values......Page 295 The Reader......Page 297 The Writer......Page 301 Taking It With You......Page 303 A Foolish Consistency is the Hobgoblin of Little Minds......Page 305 Blank Lines......Page 306 Imports......Page 307 Pet Peeves......Page 308 Other Recommendations......Page 309 Comments......Page 310 Documentation Strings......Page 311 Descriptive: Naming Styles......Page 312 Package and Module Names......Page 313 Method Names and Instance Variables......Page 314 Designing for Inheritance......Page 315 Programming Recommendations......Page 316 Copyright......Page 319 Voting Scores......Page 321 Copyright......Page 322 Easter Egg......Page 323 Copyright......Page 324 What is a Docstring?......Page 325 One-Line Docstrings......Page 326 Multi-Line Docstrings......Page 327 Handling Docstring Indentation......Page 328 Acknowledgments......Page 329 Backwards Compatibility Rules......Page 331 Making Incompatible Changes......Page 332 Copyright......Page 333 Timeline......Page 335 Compatibility and Transition......Page 336 Copyright......Page 337 Rationale......Page 339 Allowed to Change......Page 340 Copyright......Page 341 A......Page 343 B......Page 344 C......Page 345 D......Page 347 F......Page 349 I......Page 351 M......Page 353 O......Page 356 P......Page 357 R......Page 358 S......Page 359 T......Page 361 V......Page 362 W......Page 363 Acknowledgments I wouldn’t have even started this project if not for the endless encouragement from my lovely wife, Angel. She’s been my sounding board, my task manager, my copy editor and my own personal cheerleader. There’s no way I could do anything like this without her help and support. I’d also like to thank my technical reviewer, George, for everything he’s done to help me out. He’s gone above and beyond the limits of his role, helping with everything from code to grammar and even a good bit of style. After enjoying his help on Pro Django, I wouldn’t have even signed on for another book without him by my side. Lastly, I never would’ve considered a book like this if not for the wonderful community around Python. The willingness of Python programmers to open their minds and their code is, I believe, unrivaled among our peers. It’s this spirit of openness that encourages me every day, leading me to discover new things and push myself beyond the limits of what I knew yesterday. We learn by doing and by seeing what others have done. I hope that you’ll take the contents of this book and do more with it than what I’ve done. There’s no better reward for all this hard work than to see better programmers writing better code. You’ve learned the basics of Python, but how do you take your skills to the next stage? Even if you know enough to be productive, there are a number of features that can take you to the next level in Python. __Pro Python__ explores concepts and features normally left to experimentation, allowing you to be even more productive and creative. In addition to pure code concerns, Pro Python will develop your programming techniques and approaches, which will help make you a better Python programmer. Not only will this book help your code, it will also help you understand and interact with the many established Python communities, or even start your own. * Take your Python knowledge and coding skills to the next level. * Write clean, innovative code that will be respected among your peers. * Make your code do more with introspection and metaprogramming. * Design complete frameworks and libraries (two are included in the book!). ## What you’ll learn ## Who this book is for This book is for intermediate to advanced Python programmers who are looking to understand how and why Python works the way it does and how they can take their code to the next level. ## Table of Contents You’ve learned the basics of Python, but how do you take your skills to the next stage? Even if you know enough to be productive, there are a number of features that can take you to the next level in Python. Pro Python explores concepts and features normally left to experimentation, allowing you to be even more productive and creative. In addition to pure code concerns, Pro Python will develop your programming techniques and approaches, which will help make you a better Python programmer. Not only will this book help your code, it will also help you understand and interact with the many established Python communities, or even start your own. Take your Python knowledge and coding skills to the next level. Write clean, innovative code that will be respected among your peers. Make your code do more with introspection and metaprogramming. Design complete frameworks and libraries (two are included in the book!). For more information, including a link to the source code referenced in the book, please visit http://propython.com/. You ́ ve learned the basics of Python, but how do you take your skills to the next stage? Even if you know enough to be productive, there are a number of features that can take you to the next level in Python. Pro Python explores concepts and features normally left to experimentation, allowing you to be even more productive and creative. In addition to pure code concerns, Pro Python will develop your programming techniques and approaches, which will help make you a better Python programmer. Not only will this book help your code, it will also help you understand and interact with the many established Python communities, or even start your own. Take your Python knowledge and coding skills to the next level. Write clean, innovative code that will be respected among your peers. Make your code do more with introspection and metaprogramming. Design complete frameworks and libraries (two are included in the book!). For more information, including a link to the source code referenced in the book, please visit http://propython.com/