Foundations of Python 3 Network Programming, Second Edition

This second edition of Foundations of Python Network Programming targets Python 2.5 through Python 2.7, the most popular production versions of the language. Python has made great strides since Apress released the first edition of this book back in the days of Python 2.3. The advances required new chapters to be written from the ground up, and others to be extensively revised. You will learn fundamentals like IP, TCP, DNS and SSL by using working Python programs; you will also be able to familiarize yourself with infrastructure components like memcached and message queues. You can also delve into network server designs, and compare threaded approaches with asynchronous event-based solutions. But the biggest change is this edition's expanded treatment of the web. The HTTP protocol is covered in extensive detail, with each feature accompanied by sample Python code. You can use your HTTP protocol expertise by studying an entire chapter on screen scraping and you can then test lxml and BeautifulSoup against a real-world web site. The chapter on web application programming now covers both the WSGI standard for component interoperability, as well as modern web frameworks like Django. Finally, all of the old favorites from the first edition are back: E-mail protocols like SMTP, POP, and IMAP get full treatment, as does XML-RPC. You can still learn how to code Python network programs using the Telnet and FTP protocols, but you are likely to appreciate the power of more modern alternatives like the paramiko SSH2 library. If you are a Python programmer who needs to learn the network, this is the book that you want by your side. Prelim......Page 1 Contents at a Glance......Page 6 Contents......Page 7 About the Authors......Page 17 About the Technical Reviewer......Page 18 Acknowledgements......Page 19 Introduction......Page 20 The Building Blocks: Stacks and Libraries......Page 23 Application Layers......Page 26 Speaking a Protocol......Page 27 A Raw Network Conversation......Page 28 Turtles All the Way Down......Page 30 The Internet Protocol......Page 31 IP Addresses......Page 32 Routing......Page 33 Packet Fragmentation......Page 35 Learning More About IP......Page 36 UDP......Page 37 Addresses and Port Numbers......Page 38 Port Number Ranges......Page 39 Sockets......Page 41 Unreliability, Backoff, Blocking, Timeouts......Page 44 Connecting UDP Sockets......Page 47 Request IDs: A Good Idea......Page 49 Binding to Interfaces......Page 50 UDP Fragmentation......Page 52 Socket Options......Page 53 Broadcast......Page 54 When to Use UDP......Page 55 Summary......Page 56 How TCP Works......Page 57 When to Use TCP......Page 58 What TCP Sockets Mean......Page 59 A Simple TCP Client and Server......Page 60 One Socket per Conversation......Page 63 Address Already in Use......Page 64 Binding to Interfaces......Page 65 Deadlock......Page 66 Closed Connections, Half-Open Connections......Page 70 Summary......Page 71 Hostnames and Domain Names......Page 73 Socket Names......Page 74 Five Socket Coordinates......Page 75 IPv6......Page 76 Modern Address Resolution......Page 77 Asking getaddrinfo() About Services......Page 78 Asking getaddrinfo() for Pretty Hostnames......Page 79 Other getaddrinfo() Flags......Page 80 Primitive Name Service Routines......Page 81 Using getsockaddr() in Your Own Code......Page 82 Better Living Through Paranoia......Page 83 A Sketch of How DNS Works......Page 85 Why Not to Use DNS......Page 87 Why to Use DNS......Page 88 Resolving Mail Domains......Page 90 Summary......Page 92 Text and Encodings......Page 93 Network Byte Order......Page 95 Framing and Quoting......Page 97 Pickles and Self-Delimiting Formats......Page 101 XML, JSON, Etc.......Page 102 Compression......Page 103 Network Exceptions......Page 104 Handling Exceptions......Page 105 Summary......Page 107 Computer Security......Page 109 IP Access Rules......Page 110 Cleartext on the Network......Page 112 TLS Encrypts Your Conversations......Page 114 TLS Verifies Identities......Page 115 Supporting TLS in Python......Page 116 The Standard SSL Module......Page 117 Summary......Page 120 Daemons and Logging......Page 121 Our Example: Sir Launcelot......Page 122 An Elementary Client......Page 124 The Waiting Game......Page 125 Running a Benchmark......Page 128 Event-Driven Servers......Page 131 Poll vs. Select......Page 134 The Semantics of Non-blocking......Page 135 Twisted Python......Page 136 Threading and Multi-processing......Page 139 Threading and Multi-processing Frameworks......Page 142 Process and Thread Coordination......Page 144 Running Inside inetd......Page 145 Summary......Page 146 Caches, Message Queues, and Map-Reduce......Page 147 Using Memcached......Page 148 Memcached and Sharding......Page 150 Message Queues......Page 152 Using Message Queues from Python......Page 153 How Message Queues Change Programming......Page 155 Map-Reduce......Page 156 Summary......Page 158 HTTP......Page 159 URL Anatomy......Page 160 Instrumenting urllib2......Page 163 The GET Method......Page 164 Codes, Errors, and Redirection......Page 166 Payloads and Persistent Connections......Page 169 POST And Forms......Page 170 Successful Form POSTs Should Always Redirect......Page 172 REST And More HTTP Methods......Page 173 Identifying User Agents and Web Servers......Page 174 Content Type Negotiation......Page 175 Compression......Page 176 HTTP Caching......Page 177 HTTPS Encryption......Page 178 HTTP Authentication......Page 179 Cookies......Page 180 Cross-Site Scripting Attacks......Page 182 Summary......Page 183 Fetching Web Pages......Page 185 Downloading Pages Through Form Submission......Page 186 The Structure of Web Pages......Page 189 Three Axes......Page 190 Diving into an HTML Document......Page 191 Selectors......Page 195 Summary......Page 199 Web Applications......Page 201 Two Tiers......Page 202 Choosing a Web Server......Page 204 WSGI......Page 205 WSGI Middleware......Page 207 Python Web Frameworks......Page 209 URL Dispatch Techniques......Page 211 Templates......Page 212 Final Considerations......Page 213 Pure-Python Web Servers......Page 214 CGI......Page 215 mod_python......Page 216 Summary......Page 217 E-mail Composition and Decoding......Page 219 E-mail Messages......Page 220 Composing Traditional Messages......Page 222 Parsing Traditional Messages......Page 224 Parsing Dates......Page 225 Understanding MIME......Page 227 Composing MIME Attachments......Page 228 MIME Alternative Parts......Page 230 Composing Non-English Headers......Page 232 Composing Nested Multiparts......Page 233 Parsing MIME Messages......Page 235 Decoding Headers......Page 237 Summary......Page 238 E-mail Clients, Webmail Services......Page 239 Sending E-Mail......Page 243 Headers and the Envelope Recipient......Page 244 Multiple Hops......Page 245 Introducing the SMTP Library......Page 246 Error Handling and Conversation Debugging......Page 247 Getting Information from EHLO......Page 250 Using Secure Sockets Layer and Transport Layer Security......Page 252 Authenticated SMTP......Page 254 SMTP Tips......Page 255 Summary......Page 256 Connecting and Authenticating......Page 257 Obtaining Mailbox Information......Page 260 Downloading and Deleting Messages......Page 261 Summary......Page 263 IMAP......Page 265 Understanding IMAP in Python......Page 266 IMAPClient......Page 268 Message Numbers vs. UIDs......Page 270 Summary Information......Page 271 Downloading an Entire Mailbox......Page 272 Downloading Messages Individually......Page 274 Flagging and Deleting Messages......Page 279 Deleting Messages......Page 280 Searching......Page 281 Manipulating Folders and Messages......Page 282 Summary......Page 283 Command-Line Automation......Page 285 Command-Line Expansion and Quoting......Page 287 Unix Has No Special Characters......Page 288 Quoting Characters for Protection......Page 290 The Terrible Windows Command Line......Page 291 Things Are Different in a Terminal......Page 292 Terminals Do Buffering......Page 295 Telnet......Page 296 SSH: The Secure Shell......Page 300 An Overview of SSH......Page 301 SSH Host Keys......Page 302 SSH Authentication......Page 304 Shell Sessions and Individual Commands......Page 305 SFTP: File Transfer Over SSH......Page 308 Other Features......Page 311 Summary......Page 312 What to Use Instead of FTP......Page 313 Communication Channels......Page 314 Using FTP in Python......Page 315 ASCII and Binary Files......Page 316 Advanced Binary Downloading......Page 317 Uploading Data......Page 319 Advanced Binary Uploading......Page 320 Handling Errors......Page 321 Detecting Directories and Recursive Download......Page 323 Creating Directories, Deleting Things......Page 324 Summary......Page 325 RPC......Page 327 Features of RPC......Page 328 XML-RPC......Page 329 JSON-RPC......Page 335 Self-documenting Data......Page 337 Talking About Objects: Pyro and RPyC......Page 338 An RPyC Example......Page 339 RPC, Web Frameworks, Message Queues......Page 341 Binary Options: Thrift and Protocol Buffers......Page 342 Summary......Page 343 Index......Page 345 Prelim 1 Contents at a Glance 6 Contents 7 About the Authors 17 About the Technical Reviewer 18 Acknowledgements 19 Introduction 20 Introduction to Client/Server Networking 23 The Building Blocks: Stacks and Libraries 23 Application Layers 26 Speaking a Protocol 27 A Raw Network Conversation 28 Turtles All the Way Down 30 The Internet Protocol 31 IP Addresses 32 Routing 33 Packet Fragmentation 35 Learning More About IP 36 UDP 37 Should You Read This Chapter? 38 Addresses and Port Numbers 38 Port Number Ranges 39 Sockets 41 Unreliability, Backoff, Blocking, Timeouts 44 Connecting UDP Sockets 47 Request IDs: A Good Idea 49 Binding to Interfaces 50 UDP Fragmentation 52 Socket Options 53 Broadcast 54 When to Use UDP 55 Summary 56 TCP 57 How TCP Works 57 When to Use TCP 58 What TCP Sockets Mean 59 A Simple TCP Client and Server 60 One Socket per Conversation 63 Address Already in Use 64 Binding to Interfaces 65 Deadlock 66 Closed Connections, Half-Open Connections 70 Using TCP Streams like Files 71 Summary 71 Socket Names and DNS 73 Hostnames and Domain Names 73 Socket Names 74 Five Socket Coordinates 75 IPv6 76 Modern Address Resolution 77 Asking getaddrinfo() Where to Bind 78 Asking getaddrinfo() About Services 78 Asking getaddrinfo() for Pretty Hostnames 79 Other getaddrinfo() Flags 80 Primitive Name Service Routines 81 Using getsockaddr() in Your Own Code 82 Better Living Through Paranoia 83 A Sketch of How DNS Works 85 Why Not to Use DNS 87 Why to Use DNS 88 Resolving Mail Domains 90 Zeroconf and Dynamic DNS 92 Summary 92 Network Data and Network Errors 93 Text and Encodings 93 Network Byte Order 95 Framing and Quoting 97 Pickles and Self-Delimiting Formats 101 XML, JSON, Etc. 102 Compression 103 Network Exceptions 104 Handling Exceptions 105 Summary 107 TLS and SSL 109 Computer Security 109 IP Access Rules 110 Cleartext on the Network 112 TLS Encrypts Your Conversations 114 TLS Verifies Identities 115 Supporting TLS in Python 116 The Standard SSL Module 117 Loose Ends 120 Summary 120 Server Architecture 121 Daemons and Logging 121 Our Example: Sir Launcelot 122 An Elementary Client 124 The Waiting Game 125 Running a Benchmark 128 Event-Driven Servers 131 Poll vs. Select 134 The Semantics of Non-blocking 135 Event-Driven Servers Are Blocking and Synchronous 136 Twisted Python 136 Load Balancing and Proxies 139 Threading and Multi-processing 139 Threading and Multi-processing Frameworks 142 Process and Thread Coordination 144 Running Inside inetd 145 Summary 146 Caches, Message Queues, and Map-Reduce 147 Using Memcached 148 Memcached and Sharding 150 Message Queues 152 Using Message Queues from Python 153 How Message Queues Change Programming 155 Map-Reduce 156 Summary 158 HTTP 159 URL Anatomy 160 Relative URLs 163 Instrumenting urllib2 163 The GET Method 164 The Host Header 166 Codes, Errors, and Redirection 166 Payloads and Persistent Connections 169 POST And Forms 170 Successful Form POSTs Should Always Redirect 172 POST And APIs 173 REST And More HTTP Methods 173 Identifying User Agents and Web Servers 174 Content Type Negotiation 175 Compression 176 HTTP Caching 177 The HEAD Method 178 HTTPS Encryption 178 HTTP Authentication 179 Cookies 180 HTTP Session Hijacking 182 Cross-Site Scripting Attacks 182 WebOb 183 Summary 183 Screen Scraping 185 Fetching Web Pages 185 Downloading Pages Through Form Submission 186 The Structure of Web Pages 189 Three Axes 190 Diving into an HTML Document 191 Selectors 195 Summary 199 Web Applications 201 Web Servers and Python 202 Two Tiers 202 Choosing a Web Server 204 WSGI 205 WSGI Middleware 207 Python Web Frameworks 209 URL Dispatch Techniques 211 Templates 212 Final Considerations 213 Pure-Python Web Servers 214 CGI 215 mod_python 216 Summary 217 E-mail Composition and Decoding 219 E-mail Messages 220 Composing Traditional Messages 222 Parsing Traditional Messages 224 Parsing Dates 225 Understanding MIME 227 How MIME Works 228 Composing MIME Attachments 228 MIME Alternative Parts 230 Composing Non-English Headers 232 Composing Nested Multiparts 233 Parsing MIME Messages 235 Decoding Headers 237 Summary 238 SMTP 239 E-mail Clients, Webmail Services 239 How SMTP Is Used 243 Sending E-Mail 243 Headers and the Envelope Recipient 244 Multiple Hops 245 Introducing the SMTP Library 246 Error Handling and Conversation Debugging 247 Getting Information from EHLO 250 Using Secure Sockets Layer and Transport Layer Security 252 Authenticated SMTP 254 SMTP Tips 255 Summary 256 POP 257 Compatibility Between POP Servers 257 Connecting and Authenticating 257 Obtaining Mailbox Information 260 Downloading and Deleting Messages 261 Summary 263 IMAP 265 Understanding IMAP in Python 266 IMAPClient 268 Examining Folders 270 Message Numbers vs. UIDs 270 Message Ranges 271 Summary Information 271 Downloading an Entire Mailbox 272 Downloading Messages Individually 274 Flagging and Deleting Messages 279 Deleting Messages 280 Searching 281 Manipulating Folders and Messages 282 Asynchrony 283 Summary 283 Telnet and SSH 285 Command-Line Automation 285 Command-Line Expansion and Quoting 287 Unix Has No Special Characters 288 Quoting Characters for Protection 290 The Terrible Windows Command Line 291 Things Are Different in a Terminal 292 Terminals Do Buffering 295 Telnet 296 SSH: The Secure Shell 300 An Overview of SSH 301 SSH Host Keys 302 SSH Authentication 304 Shell Sessions and Individual Commands 305 SFTP: File Transfer Over SSH 308 Other Features 311 Summary 312 FTP 313 What to Use Instead of FTP 313 Communication Channels 314 Using FTP in Python 315 ASCII and Binary Files 316 Advanced Binary Downloading 317 Uploading Data 319 Advanced Binary Uploading 320 Handling Errors 321 Detecting Directories and Recursive Download 323 Creating Directories, Deleting Things 324 Doing FTP Securely 325 Summary 325 RPC 327 Features of RPC 328 XML-RPC 329 JSON-RPC 335 Self-documenting Data 337 Talking About Objects: Pyro and RPyC 338 An RPyC Example 339 RPC, Web Frameworks, Message Queues 341 Recovering From Network Errors 342 Binary Options: Thrift and Protocol Buffers 342 Summary 343 Index 345 Brandon Rhodes And John Goerzen. Includes Index.