Python and XML [XML processing with Python

I found the entire book easy to read and understand. I've been programming for a while now, but am new to both Python and XML and this book was at the perfect level for me. The book is very readable and the examples used were both concise and pertinent. The Python code used in the book was easy to follow even though I've only been using Python for a few weeks. This book is mostly a tutorial on what XML is, how to manipulate XML documents using Python, and how to use XML to move data over the internet via HTTP. It was a very good practical overview of XML in the context of Python. While it wasn't all-encompassing, the book gave you just the right amount of information to get started, without burying you with details that might be better left to more of a reference style book. I almost gave it just 4 stars only because the chapter on SOAP was quite dated. Since the book was last published when SOAP was in it's infancy, I didn't find that the information provided was quite as useful as it could be. The overview of SOAP was OK, but the sample code seemed to not have the same flow as the rest of the book. That said, if you're looking for information on SOAP specifically, this book may not be your best choice. Interestingly though, the last chapter that ties everything together with a more involved example web application, shows how to do a SOAP-like web service without actually using SOAP - which actually seemed to be a lot simpler in my opinion. I would have also appreciated a little more coverage of XML Schemas, but after reading the book, I now at least know what to look for. All-in-all though, an excellent book to get you started if you're looking to work with XML documents using Python. Dedication 3 Preface 3 Audience 4 Organization 5 Conventions Used in This Book 7 How to Contact Us 7 Acknowledgments 8 Chapter 1. Python and XML 9 1.1 Key Advantages of XML 10 1.1.1 Application Neutrality 10 1.1.2 Hierarchical Structure 12 1.1.3 Platform Neutrality 13 1.1.4 International Language Support 13 1.2 The XML Specifications 15 1.2.1 XML 1.0 Recommendation 15 1.2.2 Namespaces in XML 16 1.2.3 XML as a Foundation 16 1.3 The Power of Python and XML 17 1.3.1 Python Tools for XML 20 1.3.2 The SAX and DOM APIs 20 Example 1-1. bookhandler.py 21 Example 1-2. dombook.py 23 1.3.3 More Ways to Extract Information 24 1.4 What Can We Do with It? 24 Chapter 2. XML Fundamentals 25 2.1 XML Structure in a Nutshell 25 2.2 Document Types and Schemas 26 2.2.1 Document Type Definitions 26 2.2.2 Alternate Schema Languages 27 2.2.2.1 XML Schema 27 2.2.2.2 TREX 28 2.2.2.3 RELAX-NG 28 2.2.2.4 Schematron 28 2.3 Types of Conformance 28 2.4 Physical Structures 30 2.5 Constructing XML Documents 31 2.5.1 Characters in XML Documents 31 2.5.1.1 The ASCII character set 32 2.5.1.2 The ISO-8859-1 character set 32 2.5.1.3 UTF-8 Encoding 32 2.5.2 Text, Character Data, and Markup 32 2.5.2.1 Names 34 2.5.3 Whitespace in Character Data 34 2.5.4 End-of-Line Handling 36 2.5.5 Language Identification 36 2.5.6 The Document Prolog 37 2.5.7 Start, End, and Empty Element Tags 38 2.5.7.1 Quotes around attribute values 39 2.5.8 Comments 39 2.5.9 Processing Instructions 40 2.5.10 CDATA Sections 40 2.6 Document Type Definitions 41 2.6.1 Entity Declarations 41 2.6.2 Element Type Declarations 43 2.6.2.1 Content models 44 2.6.3 Attribute Declarations 46 2.6.3.1 Attribute data types 46 2.6.3.2 Attribute values and constraints 47 2.7 Canonical XML 49 2.7.1 The Canonical XML Data Model 49 2.7.2 Document Order 50 2.7.3 Canonical XML Structure 50 2.8 Going Beyond the XML Specification 50 2.8.1 XML Namespaces 50 2.8.2 Extracting Information Using XPath 51 2.8.3 Using XLink to Link XML Documents 51 2.8.4 Communicating with XML Protocols 51 2.8.5 Replacing HTML with XHTML 52 2.8.6 Transforming XML with XSLT 52 Chapter 3. The Simple API for XML 53 3.1 The Birth of SAX 53 3.2 Understanding SAX 54 3.2.1 Using SAX in an Application 55 Figure 3-1. Components of a SAX application 55 3.2.2 SAX Handler Objects 56 3.2.2.1 ContentHandler 56 3.2.2.2 ErrorHandler 56 3.2.2.3 DTDHandler 57 3.2.2.4 EntityResolver 57 3.2.2.5 Other handler objects 58 3.2.3 SAX Reader Objects 58 3.3 Reading an Article 59 59 Example 3-1. article.xml 59 3.3.1 Writing a Simple Handler 59 3.3.2 Creating the Main Program 60 Example 3-2. art.py 60 3.3.3 Adding Intelligence 60 Example 3-3. Enhanced ArticleHandler 62 3.3.4 Using the Additional Information 62 3.4 Searching File Information 63 3.4.1 Creating the Index Generator 63 3.4.1.1 Creating the IndexFile class 64 Example 3-4. index.py 66 3.4.1.2 Running index.py 68 3.4.2 Searching the Index 69 Example 3-5. saxfinder.py 70 3.5 Building an Image Index 71 3.5.1 Creating Thumbnail Images 72 3.5.1.1 Creating thumbnails on Windows 73 3.5.2 Implementing the SAXThumbs Handler 73 Example 3-6. saxthumbs.py 74 Example 3-7. thumbmaker.py 75 3.5.3 Viewing Your Thumbnails 76 3.6 Converting XML to HTML 76 77 Example 3-8. genxml.py 77 3.6.1 The Generated Document 79 Figure 3-2. genhtml.py output in a browser 80 3.6.2 The Conversion Handler 80 Example 3-9. handlers.py 81 3.6.3 Driving the Conversion Handler 81 Example 3-10. genhtml.py 82 3.7 Advanced Parser Factory Usage 82 3.8 Native Parser Interfaces 83 3.8.1 Using PyExpat Directly 83 Example 3-11. genhtml2.py with PyExpat 85 Chapter 4. The Document Object Model 85 4.1 The DOM Specifications 86 4.1.1 Levels of the Specification 87 4.1.2 Feature Specifications 88 4.2 Understanding the DOM 89 90 Figure 4-1. A simple DOM hierarchy 90 4.3 Python DOM Offerings 90 4.3.1 Streamlining with Minidom 91 4.3.2 Using Pulldom 91 4.3.3 4DOM: A Full Implementation 91 4.4 Retrieving Information 92 4.4.1 Getting a Document Object 92 4.4.1.1 Loading a document using 4DOM 92 4.4.1.2 Loading a document using minidom 92 4.4.2 Determining a Node's Type 93 4.4.3 Getting a Node's Children 93 4.4.4 Getting a Node's Siblings 94 4.4.5 Extracting Elements by Name 95 Example 4-1. po.xml 95 Example 4-2. po.py 96 4.4.6 Examining NodeList Members 96 Example 4-3. textme.py 96 4.4.7 Looking at Attributes 97 4.5 Changing Documents 98 4.5.1 Creating New Nodes 98 4.5.2 Adding and Moving Nodes 99 4.5.3 Removing Nodes 99 Example 4-4. domit.py 99 4.5.4 Changing a Document's Structure 101 Example 4-5. domit2.py 101 4.6 Building a Web Application 102 4.6.1 Preparing the Web Server 102 4.6.1.1 Ensuring the script's execution 103 4.6.1.2 Enabling write permission 103 4.6.2 The Web Application Structure 104 Figure 4-2. The site architecture 104 4.6.2.1 The Article class 104 Example 4-6. Article class from article.py 106 4.6.2.2 The Storage class 107 Example 4-7. storage.py 108 4.6.3 Implementing Site Logic 109 4.6.3.1 The ArticleManager class 109 Example 4-8. ArticleManager.py 111 4.6.4 Controlling the Application 112 Example 4-9. start.cgi 114 4.7 Going Beyond SAX and DOM 115 Chapter 5. Querying XML with XPath 115 5.1 XPath at a Glance 116 5.2 Where Is XPath Used? 116 5.3 Location Paths 116 5.3.1 An Example Document 117 Example 5-1. ships.xml 117 5.3.2 A Path Hosting Script 118 Example 5-2. xp.py 118 5.3.3 Getting Character Data 119 5.3.4 Specifying an Index 119 5.3.5 Testing Descendent Nodes 120 5.3.6 Testing Attributes 121 5.3.7 Selecting Elements 122 5.3.8 Additional Operators 122 5.4 XPath Arithmetic Operators 123 123 Example 5-3. products.xml 123 Example 5-4. products.xsl 123 5.5 XPath Functions 124 5.5.1 Working with Numbers 124 Example 5-5. products.xsl 125 Figure 5-1. Using the sum( ) XPath function 125 5.5.2 Working with Strings 126 5.5.3 Working with Nodes 127 5.6 Compiling XPath Expressions 128 129 Example 5-6. xp.py 129 Chapter 6. Transforming XML with XSLT 129 6.1 The XSLT Specification 130 130 Figure 6-1. The XSLT transformation process 130 6.2 XSLT Processors 130 6.3 Defining Stylesheets 131 6.3.1 Simplified Stylesheets 132 Example 6-1. ships-template.html 132 Example 6-2. ships.html 133 Figure 6-2. ships.html in a browser 134 6.3.2 Standalone Stylesheets 134 Example 6-3. ships.xsl 134 Example 6-4. ships2.html 135 6.3.3 Embedded Stylesheets 136 6.4 Using XSLT from the Command Line 136 6.5 XSLT Elements 137 6.5.1 The Stylesheet Element 137 6.5.2 Creating a Template Element 138 6.5.3 Applying Templates 139 6.5.4 Getting the Value of a Node 142 6.5.5 Iterating over Elements 143 6.6 A More Complex Example 144 6.6.1 File Template 144 6.6.2 Class Template 145 6.6.3 Method Template 145 Example 6-5. pyxml.xsl 146 6.7 Embedding XSLT Transformations in Python 147 6.7.1 Creating the Source XML 147 6.7.2 Creating a Simple Stylesheet 147 Example 6-6. story.xsl 148 Figure 6-3. Transformation using a simple stylesheet 148 6.7.3 Creating a Stylesheet with Edit Functions 149 Example 6-7. edstory.xsl 149 Figure 6-4. Editing the XML inside a web browser 150 6.7.4 Creating the CGI Script 150 6.7.5 Selecting a Mode 151 Example 6-8. xslt.cgi 152 6.8 Choosing a Technique 153 Chapter 7. XML Validation and Dialects 153 7.1 Working with DTDs 154 7.1.1 Validating with the Internal DTD Subset 154 Example 7-1. product.xml with a bad product element 154 7.1.2 Validating with an External DTD Subset 156 Example 7-2. order.xml with an external DTD 156 Example 7-3. order.dtd 156 7.2 Validation at Runtime 157 158 Example 7-4. A BadOrderErrorHandler class implements ErrorHandler in xpHandlers.py 158 Example 7-5. A DTDHandler class implements DTDConsumer in xpHandlers.py 158 Example 7-6. Command-line validator (val.py) 159 7.3 The BillSummary Example 159 7.3.1 The Flat File 160 Example 7-7. BillSummary.txt 161 7.3.2 The Web Form 161 Example 7-8. The web form flatfile.html will post your flat file 162 Figure 7-1. A web form hosts a flat text file 163 7.3.3 Starting the CGI 164 Example 7-9. flatfile.cgi, a first step version of the CGI 164 Figure 7-2. Base functionality of the CGI script 165 7.3.4 Conversion and Validation 165 7.3.4.1 Converting text to XML 165 Example 7-10. FlatfileParser.py 168 7.3.4.2 Validating the XML 169 Example 7-11. A well-formed, converted, valid, BillSummary.xml 169 Example 7-12. BillSummary.dtd 170 7.3.4.3 Creating a validation handler 171 Example 7-13. ValidityError.py 171 7.3.5 Completing the CGI 172 7.3.5.1 Defining success and error functions 172 7.3.5.2 Converting the flat file to XML 173 7.3.5.3 Validating the converted XML 174 7.3.5.4 Displaying the XML 175 Example 7-14. flat2xml.cgi 176 7.3.6 Running the Application in a Browser 178 Figure 7-3. A successful run of flat2xml.cgi 178 Figure 7-4. A run of flat2xml.cgi with excessive validation errors 178 7.4 Dialects, Frameworks, and Workflow 180 7.5 What Does ebXML Offer? 180 7.5.1 ebXML Document Structure 180 7.5.2 Business Process and Modeling 181 7.5.3 Phases of ebXML 181 Chapter 8. Python Internet APIs 182 8.1 Connecting Web Sites 182 8.1.1 Continuing Improvement 182 8.1.2 Python to the Rescue 183 8.2 Working with URLs 183 8.2.1 Encoding URLs 184 8.2.2 Quoting URLs 185 8.2.3 Unquoting URLs 185 8.3 Opening URLs 185 8.3.1 Using FTP 186 8.3.2 Retrieving URLs 186 Example 8-1. retrieve.py 187 8.4 Connecting with HTTP 188 8.4.1 HTTP Conversations 188 8.4.2 Request Types 189 8.4.3 Getting a Document with Python 190 Example 8-2. Making an HTTP Request 191 8.4.4 Building a Query String with httplib 192 8.4.5 Baking Cookies for the Server 192 8.4.6 Performing a POST Operation 193 8.4.6.1 Creating a POST catcher 193 Example 8-3. favquote.cgi 193 8.4.6.2 Ensuring proper URL encoding 193 8.4.6.3 Performing a POST with httplib 194 8.4.6.4 Illustrating a complete POST operation 195 Example 8-4. post.py 195 8.5 Using the Server Classes 196 8.5.1 BaseHTTPServer Module Classes 197 8.5.2 Server Core Concepts 198 8.5.2.1 Instantiating a server class 198 8.5.2.2 Serving a GET 198 8.5.2.3 Serving a POST 199 8.5.3 Building a Complete Server 200 Example 8-5. HTTPServer.py 200 8.5.3.1 Running a GET request 202 Figure 8-1. A browser connecting to the server with a GET request. 202 8.5.3.2 Running a POST request 203 Example 8-6. testServer.html 203 Figure 8-2. A web form to test the server 203 Figure 8-3. A response to a submitted form 204 Chapter 9. Python, Web Services, and SOAP 205 9.1 Python Web Services Support 206 9.2 The Emerging SOAP Standard 206 9.2.1 SOAP Messages 206 9.2.2 Exchanging SOAP Messages 207 9.2.3 Encoding SOAP Messages 208 9.2.4 Constructing SOAP Envelopes 208 9.2.4.1 SOAP packet requirements 208 9.2.4.2 SOAP encoding style 209 9.2.5 Using SOAP Headers 209 9.2.6 SOAP Body Elements 210 9.2.7 Error Message and SOAP Fault 210 9.2.7.1 Fault element 210 9.2.7.2 Fault codes 211 9.2.8 SOAP Encoding Techniques 211 9.2.9 SOAP Encoding Rules 212 9.2.10 Simple Types 213 9.2.11 Compound Types 213 9.2.12 SOAP over HTTP 213 9.2.12.1 The SOAPAction header 214 9.2.12.2 SOAP HTTP responses 214 9.2.13 SOAP for RPC 214 9.3 Python SOAP Options 214 9.3.1 Working with SOAPy 215 9.3.2 Working with MSSOAP 216 9.3.3 MSSOAP Serialization Basics 216 9.3.3.1 Adding URIs and namespaces 216 9.3.3.2 Creating the SOAP envelope 217 9.3.3.3 Making the call 218 9.4 Example SOAP Server and Client 218 9.4.1 Requirements for Using MSSOAP 218 9.4.1.1 Getting Microsoft SOAP Toolkit 2.0 219 9.4.1.2 Making the samples web-visible 219 9.4.1.3 Getting Python COM support 219 9.4.1.4 Fixing MSSOAP with makepy.py 220 Figure 9-1. Selecting the SOAP Type Library with makepy.py 220 9.4.2 Server Setup 221 9.4.3 A Python SOAP Client 221 9.4.3.1 Defining reusable basics 222 Example 9-1. PyCalcSerial.py 224 9.5 What About XML-RPC? 226 Chapter 10. Python and Distributed Systems Design 226 10.1 Sample Application and Flow Analysis 227 227 Figure 10-1. The sample distributed application 227 10.1.1 Decoupling Application Systems 227 10.1.2 Routing Adds Flexibility 228 10.1.3 Routing Adds Scalability 228 10.2 Understanding the Scope 228 10.3 Building the Database 229 10.3.1 Creating a Profiles Database 229 10.3.2 Creating a Customer Table 230 10.3.3 Populating the Database 230 Example 10-1. popdb.py 231 10.4 Building the Profiles Access Class 232 10.4.1 The Interfaces 233 10.4.2 Getting Profiles 234 10.4.2.1 Connecting with the database 235 10.4.2.2 Building the XML document 235 10.4.2.3 Returning a DOM instead of a string 237 10.4.3 Inserting and Deleting Profiles 237 10.4.3.1 Inserting a profile 238 10.4.3.2 Deleting a profile 240 10.4.4 Updating Profiles 240 10.4.5 The Complete CustomerProfile Class 241 Example 10-2. CustomerProfile.py 242 10.5 Creating an XML Data Store 245 10.5.1 A Large XML File 246 Example 10-3. OfferXMLStore.xml 246 10.5.2 Creating an XML Access Object 247 10.5.2.1 The interfaces 247 10.5.2.2 Using the XMLOffer class 248 10.5.2.3 Creating the XMLOffer class 249 10.5.2.3.1 Retrieval methods 249 10.5.2.3.2 Modification methods 250 Example 10-4. XMLOffer.py 251 10.6 The XML Switch 253 10.6.1 XML Architecture 254 10.6.2 Core XML Switch Classes 254 10.6.3 The XMLMessage Class 255 10.6.3.1 XMLMessage format 255 Example 10-5. An example message.xml file 255 10.6.3.2 XMLMessage class 256 Example 10-6. runxm.py -- using the XMLMessage object 256 10.6.3.3 XML message code architecture 258 10.6.3.4 XMLMessage code listing 259 Example 10-7. XMLMessage.py 259 10.6.4 The XML Switch Service 261 10.6.5 The XML Switch Client 262 10.6.5.1 Using postMsg.html to send back XML 262 Example 10-8. The postMsg.html file 262 Figure 10-2. Using postMsg.html to connect to the server 263 Figure 10-3. Posting an RPC call with postMsg.html 264 10.6.5.2 Using the XSC client 264 Example 10-9. msgGetProfile.xml 265 Example 10-10. Running xcs.py from the command line 265 10.6.5.3 Using the XSC API 266 Example 10-11. xsc.py, the client to the XML Switch 266 10.6.6 The XMLSwitchHandler Server Class 268 10.6.6.1 XMLSwitchHandler code architecture 268 10.6.6.2 XMLSwitchHandler listing 270 Example 10-12. XMLSwitchHandler.py 270 10.7 Running the XML Switch 274 274 Example 10-13. The XML Switch launching script: runxs.py 274 10.8 A Web Application 275 10.8.1 Connecting to a Web Service 275 Figure 10-4. A detail of the web application 275 10.8.2 The Components 276 10.8.3 The Topology 276 10.8.4 The Code Architecture 277 Figure 10-5. intro.html is the start page for the CGI example 277 Example 10-14. intro.html 278 10.8.5 The CGI Functionality 278 10.8.5.1 Extracting profile information 280 Figure 10-6. The profile-editing form 281 10.8.5.2 Updating profile information 282 10.8.5.3 Displaying all offers 283 10.8.6 The Complete sp.py Listing 284 Example 10-15. sp.py 284 10.8.7 Running the Site as a User 288 Appendix A. Installing Python and XML Tools 288 A.1 Installing Python 289 A.1.1 Windows 289 A.1.2 Linux and Unix 289 A.2 Installing PyXML 290 A.3 Installing 4Suite 291 Appendix B. XML Definitions 292 292 292 Example B-1. gen-td.py—a script to print XML defi 292 Figure B-1. termdef.html loaded in a browser 293 B.1 XML Definitions 294 Appendix C. Python SAX API 302 Appendix D. Python DOM API 314 320 Node Constants 320 Node Properties and Methods 321 D.1 4DOM Extensions 332 Appendix E. Working with MSXML3.0 333 E.1 Setting Up MSXML3.0 333 E.2 Basic DOM Operations 333 334 Example E-1. books.xml 334 E.2.1 MSXML Nodes 335 E.2.2 Using a NodeList 335 Example E-2. people.xml 336 Example E-3. nodelists.py 336 E.3 MSXML3.0 Support for XSLT 337 E.3.1 Source XML 337 Example E-4. 1999 temps.xml 337 E.3.2 XSL Stylesheet 337 Example E-5. temps.xsl 338 E.3.3 Running an MSXML Transformation 339 Example E-6. transform.py 339 Figure E-1. The result of the transformation 340 E.4 Handling Parsing Errors 340 E.5 MSXML3.0 Reference 341 MSXML3.0 Document Object Methods 342 MSXML3.0 Document Object Properties 343 MSXML3.0 Node Object Methods 345 MSXML3.0 Node Object Properties 346 Appendix F. Additional Python XML Tools 351 F.1 Pyxie 351 F.2 Python XML Tools 353 F.3 XML Schema Validator 353 F.4 Sab-pyth 354 F.5 Redfoot 354 F.6 XML Components for Zope 354 F.6.1 Parsed XML 354 F.6.2 Page Templates 354 F.7 Online Resources 355 Colophon 355 Full Description 357 About the Author 357 you want in the servers,put it here. scan books,download books,contribute. If you are a Python programmer who wants to incorporate XML into your skill set, this is the book for you. Python has attracted a wide variety of developers, who use it either as glue to connect critical programming tasks together, or as a complete cross-platform application development language. Yet, because it is object-oriented and has powerful text manipulation abilities, Python is an ideal language for manipulating XML. Python & XML gives you a solid foundation for using these two languages together. Loaded with practical examples, this new volume highlights common application tasks, so that you can learn by doing. The book starts with the basics then quickly progresses to complex topics, like transforming XML with XSLT, querying XML with XPath, and working with XML dialects and validation. It also explores the more advanced issues: using Python with SOAP and distributed web services, and using Python to create scalable streams between distributed applications (like databases and web servers). The book provides effective practical applications, while referencing many of the tools involved in XML processing and Python, and highlights cross-platform issues along with tasks relevant to enterprise computing. You will find ample coverage of XML flow analysis and details on ways in which you can transport XML through your network. Whether you are using Python as an application language, or as an administrative or middleware scripting language, you are sure to benefit from this book. If you want to use Python to manipulate XML, this is your guide. If you are a Python programmer who wants to incorporate XML into your skill set, this is the book for you. Python has attracted a wide variety of developers, who use it either as glue to connect critical programming tasks together, or as a complete cross-platform application development language. Yet, because it is object-oriented and has powerful text manipulation abilities, Python is an ideal language for manipulating XML.Python & XML gives you a solid foundation for using these two languages together. Loaded with practical examples, this new volume highlights common application tasks, so that you can learn by doing. The book starts with the basics then quickly progresses to complex topics, like transforming XML with XSLT, querying XML with XPath, and working with XML dialects and validation. It also explores the more advanced issues: using Python with SOAP and distributed web services, and using Python to create scalable streams between distributed applications (like databases and web servers).The book provides effective practical applications, while referencing many of the tools involved in XML processing and Python, and highlights cross-platform issues along with tasks relevant to enterprise computing. You will find ample coverage of XML flow analysis and details on ways in which you can transport XML through your network.Whether you are using Python as an application language, or as an administrative or middleware scripting language, you are sure to benefit from this book. If you want to use Python to manipulate XML, this is your guide.