Learning Python Networking: A Complete Guide to Build and Deploy Strong Networking Capabilities Using Python 3. 7 and Ansible , 2nd Edition

Title Page......Page 2 Copyright and Credits......Page 3 Learning Python Networking Second Edition......Page 4 About Packt......Page 5 Why subscribe?......Page 6 Packt.com......Page 7 Contributors......Page 8 About the authors......Page 9 About the reviewers......Page 11 Packt is searching for authors like you......Page 12 Preface......Page 24 Who this book is for......Page 25 What this book covers......Page 26 To get the most out of this book......Page 28 Download the example code files......Page 29 Download the color images......Page 30 Conventions used......Page 31 Get in touch......Page 33 Reviews......Page 34 Section 1: Introduction to Network and HTTP Programming......Page 35 Network Programming with Python......Page 36 Technical requirements......Page 38 An introduction to TCP/IP networking......Page 39 Introduction to TCP/IP......Page 40 The protocol stack, layer by layer......Page 41 UDP......Page 44 TCP......Page 45 Protocol concepts and the problems that protocols solve......Page 46 IP addresses and ports......Page 47 Network interfaces......Page 49 UDP versus TCP......Page 52 DHCP......Page 54 DNS......Page 56 Addressing......Page 58 NAT......Page 59 IPv4......Page 60 IPv6......Page 62 Python network programming through libraries......Page 63 An introduction to the PyPI Python repository......Page 64 Alternatives to pip for installing packages......Page 65 Conda......Page 66 Virtualenv......Page 67 Pipenv......Page 68 An introduction to libraries for network programming with Python......Page 69 Introduction to sockets......Page 70 Socket module in Python......Page 71 Client socket methods......Page 72 Server socket methods......Page 73 Working with RFC......Page 74 Extracting RFC information......Page 76 Downloading an RFC with urllib......Page 77 Downloading an RFC with requests......Page 79 Downloading an RFC with the socket module......Page 80 Interacting with Wireshark with pyshark......Page 82 Introduction to Wireshark......Page 83 Wireshark installation......Page 85 Capturing packets with Wireshark......Page 86 Network traffic in Wireshark......Page 88 Color coding in Wireshark......Page 90 Working with filters in Wireshark......Page 91 Filtering by protocol name......Page 93 HTTP objects filter......Page 94 Capture filters......Page 95 Display filters......Page 96 Analyzing networking traffic using the pyshark library......Page 97 FileCapture and LiveCapture in pyshark......Page 101 Summary......Page 104 Questions......Page 105 Further reading......Page 106 Programming for the Web with HTTP......Page 107 Technical requirements......Page 108 Consuming web services in Python with urllib......Page 109 Status codes......Page 111 Handling exceptions......Page 112 HTTP headers......Page 113 User agent......Page 115 Customizing requests with urllib......Page 116 Getting headers with a proxy......Page 119 Content types......Page 120 Extracting links from a URL with urllib......Page 121 Getting images from a URL with urllib......Page 123 Working with URLs......Page 124 Consuming web services in Python with requests......Page 126 Introduction to requests......Page 127 Checking HTTP headers......Page 129 Proxy requests......Page 131 Get whois information......Page 132 Working with JSON......Page 133 Handling forms with urllib and requests with Python 3.7......Page 134 Handling forms with urllib......Page 135 Handling forms with requests......Page 139 Handling cookies with urllib and requests with Python......Page 141 What are cookies?......Page 142 Handling cookies with urllib......Page 144 Cookie handling with requests......Page 147 Handling HTTP Basic and Digest Authentication with requests......Page 149 Introduction to authentication mechanisms......Page 150 HTTP Basic authentication......Page 151 HTTP Digest authentication......Page 152 Summary......Page 154 Questions......Page 155 Further reading......Page 156 Section 2: Interacting with APIs, Web Scraping, and Server Scripting......Page 157 Application Programming Interface in Action......Page 158 Technical requirements......Page 159 Introduction to REST APIs......Page 160 Advantages of using REST APIs......Page 161 Introduction to JSON and the JSON module......Page 162 Encoding and decoding with the JSON package......Page 163 Using dict with JSON......Page 165 Interacting with a JSON hybrid-REST API (Twitter)......Page 166 The Twitter API......Page 167 Registering your application for the Twitter API......Page 170 Authenticating requests with OAuth......Page 171 Collecting information from Twitter......Page 172 A Twitter client......Page 174 Retrieving tweets from a timeline......Page 176 Searching tweets......Page 178 Consuming the Twitter REST API with Python......Page 180 Connecting with the Twitter API......Page 181 Accessing Twitter API resources......Page 183 Streaming APIs with Tweepy......Page 185 Introduction to XML......Page 188 Getting started with XML......Page 189 The XML APIs......Page 190 Processing XML with ElementTree......Page 191 Pretty printing......Page 193 Reading an XML file......Page 194 Working with XML and a full REST API (Amazon S3 bucket) with the Boto module......Page 197 The Amazon S3 API......Page 198 Registering with AWS......Page 199 Authentication with AWS......Page 200 S3 buckets and objects......Page 203 Creating a bucket with the S3 API......Page 205 Uploading and downloading file......Page 209 Listing buckets......Page 211 Parsing XML and handling errors......Page 213 Connecting to S3 with the Python Boto package......Page 214 Summary......Page 217 Questions......Page 218 Further reading......Page 219 Web Scraping with BeautifulSoup and Scrapy......Page 220 Technical requirements......Page 221 Introduction to web scraping......Page 222 Web content extraction......Page 223 What is web scraping?......Page 224 HTML parsers......Page 225 Parsing HTML with lxml......Page 226 Searching with XPath......Page 230 Extracting information from web pages and parsing HTML with BeautifulSoup......Page 234 BeautifulSoup introduction......Page 235 Access to elements through DOM......Page 237 Extracting labels using regex......Page 242 Handling URL exceptions and not found tags......Page 243 Introduction to Scrapy components and architecture......Page 244 What is Scrapy?......Page 245 Scrapy architecture......Page 247 XPath expressions......Page 249 Scrapy as a framework for performing web crawling processes and data analysis......Page 250 Installation of Scrapy......Page 251 Creating a project with Scrapy......Page 253 Scrapy item class......Page 254 Spiders......Page 256 Creating our spider......Page 257 Pipelines items and export formats......Page 258 Scrapy settings......Page 260 Executing Scrapy......Page 261 Scrapy execution tips and tricks......Page 262 EuroPython project......Page 263 Executing EuroPython spider......Page 267 Working with Scrapy in the cloud......Page 269 Scrapinghub......Page 270 Portia......Page 275 Start pages and link crawling......Page 279 Summary......Page 280 Questions......Page 281 Further reading......Page 282 Engaging with Email......Page 283 Technical requirements......Page 284 Introduction to email protocols......Page 285 Sending emails with SMTP through the smtplib library......Page 287 SMTP protocol......Page 288 Working with smtplib......Page 290 Sending a basic message......Page 291 Sending messages in HTML format......Page 294 Sending emails to multiple recipients......Page 295 Sending an email with attachments......Page 296 Authentication with TLS......Page 297 Establishing a connection with a Gmail SMTP server......Page 298 Using an external SMTP service......Page 300 Creating and sending an email with an attachment......Page 303 Learning the POP3 protocol and retrieving emails with poplib......Page 305 Understanding the POP3 protocol......Page 306 Introduction to poplib......Page 308 Retrieving emails with SSL......Page 310 Establishing a connection with Gmail for reading emails ......Page 314 Gmail account configuration......Page 315 Unread messages......Page 316 Manipulating and retrieving emails on the server email using IMAP with imapclient and imaplib......Page 317 IMAP protocol......Page 318 Retrieving emails with imaplib......Page 321 Retrieving emails with imapclient......Page 324 Summary......Page 326 Questions......Page 327 Further reading......Page 328 Interacting with Remote Systems......Page 329 Technical requirements......Page 330 Understanding the SSH protocol......Page 331 SSH introduction......Page 332 Using SSH to encrypt sessions......Page 334 How the SSH protocol works......Page 335 SSH service features......Page 336 Configuring the SSH protocol to make it more secure......Page 337 SSH terminals and running commands with paramiko......Page 339 Installing paramiko......Page 340 Establishing an SSH connection with paramiko......Page 341 Running commands with paramiko......Page 343 Running an interactive shell with paramiko......Page 344 SFTP with paramiko......Page 347 Paramiko alternatives......Page 349 Fabric......Page 350 Understanding the FTP protocol for transferring files......Page 352 The File Transfer Protocol......Page 353 Introduction to ftplib......Page 354 Other ftplib functions......Page 359 Inspecting FTP packets with Wireshark......Page 360 Reading and interacting with SNMP servers......Page 363 The SNMP ......Page 364 MIB – a broad base of information......Page 366 Introduction to pysnmp......Page 368 Polling information from the SNMP agent......Page 370 Reading and interacting with LDAP servers......Page 372 The LDAP protocol......Page 373 LDAP terminology......Page 374 Introduction to python-ldap......Page 376 The LDAP FreeIPA server......Page 378 Working with LDAP3......Page 380 Accessing the LDAP server......Page 381 Finding entries in LDAP......Page 384 Summary......Page 386 Questions......Page 387 Further reading......Page 388 Section 3: IP Address Manipulation and Network Automation......Page 389 Working with IP and DNS......Page 390 Technical requirements......Page 391 Principles of the IP protocol......Page 392 Resolving the IP address with the socket package......Page 393 Validating the IP address with the socket package......Page 394 Retrieving the network configuration of a local machine......Page 395 Gathering information with the netifaces package......Page 396 Using Python to manipulate IP addresses and perform CIDR calculations......Page 398 The Python ipaddress module......Page 399 Manipulating IP addresses......Page 404 IP network objects......Page 405 Subnetting in Python......Page 406 Network interface objects......Page 407 IP address objects......Page 408 Planning IP addresses for your local area network......Page 409 The dnspython module as a tool for extracting information from DNS servers......Page 411 Working with dnspython......Page 412 Determining the destination of an MX record and its preference......Page 413 Manipulating domain names......Page 414 Converting IPv4 and IPv6 addresses into their DNS reverse map names......Page 415 Inspecting the DNS client and server communication......Page 418 GeoIP lookups with pygeoip and python-geoip......Page 420 Introduction to geolocation......Page 421 Introduction to pygeoip......Page 423 Introduction to python-geoip......Page 425 The MaxMind database in Python......Page 427 Summary......Page 429 Questions......Page 430 Further reading......Page 431 Implementing IPv6 and Address Manipulation......Page 432 Technical requirements......Page 433 Learning and understanding the IPv6 protocol......Page 434 The IPv6 protocol......Page 435 IPv6 addresses......Page 437 Representation of IPv6 addresses......Page 438 Reserved IPv6 addresses......Page 439 First steps with IPv6 – link-local......Page 440 Create an echo client and server with IPv6......Page 442 Working with sockets......Page 443 The socket server......Page 444 The socket client......Page 447 Executing client and server......Page 449 Understanding netifaces module for checking IPv6 support on your network......Page 451 Introduction to netifaces......Page 452 Other packages for getting interfaces......Page 455 Using the netaddr module as a network-address manipulation library for Python......Page 458 Operating with IPv6......Page 459 Understand ipaddress module as IPv4 and IPv6 manipulation library......Page 461 The Python ipaddress module......Page 462 IP network objects......Page 465 Subnetting in Python with IPv6......Page 466 Network interface objects......Page 467 The IP address objects......Page 468 Planning IP addresses for your local area network......Page 469 Summary......Page 471 Questions......Page 472 Further reading......Page 473 Performing Network Automation with Python and Ansible......Page 474 Technical requirements......Page 475 Basics of Ansible......Page 476 Ansible introduction......Page 477 Installing Ansible......Page 479 Configuring Ansible......Page 481 Using Ansible......Page 482 Ansible's components and architecture......Page 483 Ansible's architecture......Page 484 Ansible's inventory file......Page 487 Automating network Python tasks with Ansible......Page 489 Ansible tasks......Page 490 Ad-hoc commands......Page 491 Using playbooks......Page 492 Writing Ansible modules with Python......Page 499 Introduction to Ansible modules......Page 500 Implementing Ansible modules with Python......Page 502 Summary......Page 504 Questions......Page 505 Further reading......Page 506 Section 4: Sockets and Server Programming......Page 507 Programming with Sockets......Page 508 Technical requirements......Page 509 Basics of sockets......Page 510 Sockets introduction......Page 511 Socket types......Page 513 Getting information about ports, protocols, and domains......Page 514 Creating a TCP client......Page 516 Banner grabbing with the socket module......Page 518 Port scanning with sockets......Page 520 Inspecting the client and server communication......Page 521 Working with UDP and TCP sockets in Python 3.7......Page 524 Introduction to the TCP and UDP protocols......Page 525 Starting network programming with Python......Page 526 TCP sockets......Page 527 Starting a client......Page 529 Capturing packets in a loopback interface......Page 531 Inspecting the client and server interaction......Page 532 Code limitations......Page 533 Creating a simple UDP client and UDP server......Page 534 Implementing the UDP server......Page 535 Implementing the UDP client......Page 536 Working with IPv6 sockets in Python 3.7......Page 538 Implementing the IPv6 server......Page 539 Implementing the IPv6 client......Page 542 Executing client and server......Page 544 Non-blocking and asynchronous socket I/O......Page 546 Introducing non-blocking I/O......Page 547 The client-server model with multiple connections......Page 548 HTTPS and securing sockets with TLS......Page 552 Implementing the SSL client......Page 553 Inspecting standard SSL client and server communication......Page 558 Summary......Page 562 Questions......Page 563 Further reading......Page 564 Designing Servers and Asynchronous Programming......Page 565 Technical requirements......Page 566 Building a multiprocessing-based TCP server......Page 567 Introducing the concurrent.futures module......Page 568 Application for checking websites......Page 571 The multiprocessing approach......Page 573 Building asynchronous applications with asyncio and aiohttp......Page 575 Introducing asyncio......Page 576 Using asyncio......Page 577 Introducing event loops......Page 578 Futures......Page 579 Task manipulation with asyncio......Page 582 Downloading files with asyncio......Page 584 Introducing aiohttp......Page 586 Downloading files with aiohttp......Page 589 Other event loop solutions......Page 591 Building asynchronous network applications with Tornado......Page 592 Introducing Tornado......Page 593 Implementing the Tornado web server......Page 594 Implementing an asynchronous client with AsyncHTTPClient......Page 596 Asynchronous generators......Page 599 Utilities in Tornado for asynchronous network operations......Page 601 Building asynchronous network applications with Twisted......Page 603 Introduction to Twisted......Page 604 Protocols......Page 605 Building a basic Twisted server......Page 606 Factory......Page 608 Reactor......Page 609 Building a socket client......Page 610 Executing the client and server......Page 611 Building a Twisted client......Page 612 Building a Twisted web server......Page 614 Building asynchronous network applications with Celery......Page 615 Celery architecture......Page 616 Installing Celery......Page 617 Installing Redis......Page 618 Introduction to Redis......Page 619 Distributing Python with Celery and Redis......Page 621 Summary......Page 626 Questions......Page 627 Further reading......Page 628 Designing Applications on the Web......Page 629 Technical requirements......Page 630 Writing a web application with WSGI......Page 631 Introducing WSGI......Page 632 Creating a WSGI application......Page 634 Existing web application frameworks (Django, Flask, and Plone)......Page 637 Web frameworks......Page 638 The MVC pattern and dynamic web programming with Python......Page 641 The MVC pattern......Page 642 Dynamic web pages......Page 644 Processing dynamic pages......Page 645 Accessing a database......Page 647 Django introduction......Page 649 Creating a Django application......Page 651 Creating RESTful web applications and working with Flask and HTTP requests......Page 656 Introducing Flask......Page 657 Routing in Flask......Page 660 Jinja2 templating......Page 662 POST parameters with Flask......Page 665 Other templating engines......Page 667 Flask extensions......Page 668 Working with a database in Flask with SQLAlchemy......Page 669 Introducing SQLAlchemy......Page 670 Creating a session and ORM queries......Page 674 Using Flask with SQLAlchemy......Page 677 Summary......Page 680 Questions......Page 681 Further reading......Page 682 Assessment......Page 683 Chapter 1, Network Programming with Python......Page 684 Chapter 2, Programming for the Web with HTTP......Page 685 Chapter 3, Application Programming Interface in Action......Page 686 Chapter 4, Web Scraping with BeautifulSoup and Scrapy......Page 687 Chapter 5, Engaging with Email......Page 688 Chapter 6, Interacting with Remote Systems......Page 689 Chapter 7, Working with IP and DNS......Page 690 Chapter 8, Implementing IPv6 and Address Manipulation......Page 691 Chapter 9, Performing Network Automation with Python and Ansible......Page 692 Chapter 10, Programming with Sockets......Page 693 Chapter 11, Designing Servers and Asynchronous Programming......Page 694 Chapter 12, Designing Applications on the Web......Page 695 Another Book You May Enjoy......Page 696 Leave a review - let other readers know what you think......Page 697 Achieve improved network programmability and automation by leveraging powerful network programming concepts, algorithms, and toolsKey Features Deal with remote network servers using SSH, FTP, SNMP and LDAP protocols. Design multi threaded and event-driven architectures for asynchronous servers programming. Leverage your Python programming skills to build powerful network applications Book DescriptionNetwork programming has always been a demanding task. With full-featured and well-documented libraries all the way up the stack, Python makes network programming the enjoyable experience it should be.Starting with a walk through of today's major networking protocols, through this book, you'll learn how to employ Python for network programming, how to request and retrieve web resources, and how to extract data in major formats over the web. You will utilize Python for emailing using different protocols, and you'll interact with remote systems and IP and DNS networking. You will cover the connection of networking devices and configuration using Python 3.7, along with cloud-based network management tasks using Python.As the book progresses, socket programming will be covered, followed by how to design servers, and the pros and cons of multithreaded and event-driven architectures. You'll develop practical clientside applications, including web API clients, email clients, SSH, and FTP. These applications will also be implemented through existing web application frameworks.What you will learn Execute Python modules on networking tools Automate tasks regarding the analysis and extraction of information from a network Get to grips with asynchronous programming modules available in Python Get to grips with IP address manipulation modules using Python programming Understand the main frameworks available in Python that are focused on web application Manipulate IP addresses and perform CIDR calculations Who this book is forIf you're a Python developer or a system administrator with Python experience and you're looking to take your first steps in network programming, then this book is for you. If you're a network engineer or a network professional aiming to be more productive and efficient in networking programmability and automation then this book would serve as a useful resource. Basic knowledge of Python is assumed.Table of Contents Network Programming with Python Programming for the Web with HTTP Application Programming Interface in Action Web Scraping with BeautifulSoup and Scrapy Engaging with Email Interacting with Remote Systems Working with IP and DNS Implementing IPv6 and Address manipulation Performing Network Automation with Python and Ansible Programming with Sockets Designing Servers and Asynchronous programming Designing Applications on the Web "Network programming has always been a demanding task. With full-featured and well-documented libraries all the way up the stack, Python makes network programming the enjoyable experience it should be. Starting with a walk through of today's major networking protocols, through this book, you'll learn how to employ Python for network programming, how to request and retrieve web resources, and how to extract data in major formats over the web. You will utilize Python for emailing using different protocols, and you'll interact with remote systems and IP and DNS networking. You will cover the connection of networking devices and configuration using Python 3.7, along with cloud-based network management tasks using Python. As the book progresses, socket programming will be covered, followed by how to design servers, and the pros and cons of multithreaded and event-driven architectures. You'll develop practical clientside applications, including web API clients, email clients, SSH, and FTP. These applications will also be implemented through existing web application frameworks."--Provided by publisher