Extending and Embedding Perl

After being a Perl programmer for a while, you'll probably want to go deeper than that and perhaps enable some features that can not be done through the pure Perl code. That's when you'll need XS; the nasty, weird and hard thing that glues the external code with Perl code. While there are some tutorials and perldocs on this subject, this books seems to be the most friendly and comprehensive for the XS beginner.Extending and Embedding Perl explains how to expand the functionality and usefulness of the Perl programming language and how to use Perl from C programs. It begins simply but also covers complex issues using real code examples from the Perl source. The book discusses how to write interfaces to C libraries (as well as C++ and Fortran libraries). It shows you how to implement Perl callbacks for C libraries, how to pass Perl hashes and arrays between Perl and C, and how to use the Perl Data Language infrastructure to improve the speed of array operations.Additionally, the book peers under the hood to see how the Perl programming language really works by looking at the interpreter. The make-up of Perl variables is discussed along with details on how a Perl program is parsed and converted to executable code.what's inside• C For The Perl Programmer• Basic and Advanced XS• Embedding Perl in C Programs• Perl Internals• An API reference for the internal C interface to Perl• A reference on the typemap system• Embedding Perl into the mutt mail reader contents 7 preface 13 acknowledgments 15 about this book 16 author online 19 about the cover illustration 20 C for Perl programmers 21 1.1 Hello, world 21 1.2 The C compiler 22 1.3 Header files 23 1.4 The main function 24 1.5 Variables and functions 26 1.5.1 Function parameters 26 1.5.2 Automatic variables 27 1.5.3 Global variables 28 1.5.4 Static variables 29 1.6 Data types 30 1.6.1 C types 31 1.6.2 Types defined in Perl 35 1.7 Casting 36 1.8 Control constructs 37 1.8.1 Statements and blocks 37 1.8.2 The break and continue statements 38 1.8.3 The switch statement 39 1.9 Macros and the C preprocessor 40 1.10 Library functions 43 1.11 Summary 43 Extending Perl: an introduction 44 2.1 Perl modules 44 2.1.1 Module distributions 46 2.2 Interfacing to another language: C from XS 50 2.2.1 The Perl module 50 2.2.2 The XS file 51 2.2.3 Example: “Hello, world” 52 2.2.4 Return values 56 2.2.5 Arguments and return values 57 2.3 XS and C: taking things further 58 2.3.1 Modifying input variables 58 2.3.2 Output arguments 59 2.3.3 Compiler constants 60 2.4 What about Makefile.PL? 64 2.4.1 It really is a Perl program 67 2.5 Interface design: part 1 67 2.5.1 Status and multiple return arguments 68 2.5.2 Don’t supply what is already known 68 2.5.3 Don’t export everything 69 2.5.4 Use namespaces 69 2.5.5 Use double precision 69 2.6 Further reading 70 2.7 Summary 70 Advanced C 71 3.1 Arrays 71 3.2 Pointers 73 3.2.1 Pointers and arrays 75 3.2.2 Pointers to functions 77 3.3 Strings 78 3.3.1 Arrays of strings 79 3.4 Structures 80 3.5 File I/O 82 3.6 Memory management 83 3.6.1 Allocating memory at runtime 84 3.6.2 Altering the size of memory 85 3.6.3 Manipulating memory 85 3.6.4 Memory manipulation and Perl 87 3.7 C Traps for the Perl programmer 88 3.8 Further reading 89 3.9 Summary 89 Perl’s variable types 90 4.1 General concepts 90 4.1.1 Reference counting 91 4.1.2 Looking inside: Devel::Peek 91 4.1.3 The flag system 92 4.2 Scalar variables 94 4.2.1 The SvNULL type 94 4.2.2 SvRV: references 96 4.2.3 SvPV: string values 96 4.2.4 SvPVIV: integers 98 4.2.5 SvPVNV: floating-point numbers 99 4.2.6 SvIV and SvNV 100 4.2.7 SvOOK: offset strings 100 4.3 Magic variables: SvPVMG 101 4.4 Array variables 105 4.5 Hashes 107 4.6 Globs 111 4.7 Namespaces and stashes 114 4.8 Lexical “my” variables 115 4.9 Code blocks 116 4.9.1 Important CV flags 117 4.10 Further reading 119 4.11 Summary 119 The Perl 5 API 120 5.1 Sample entry 121 5.2 SV functions 121 5.2.1 Special SVs 121 5.2.2 Creating SVs 123 5.2.3 Accessing data 130 5.2.4 Manipulating data 139 5.2.5 String functions 144 5.2.6 References 149 5.3 AV functions 152 5.3.1 Creation and destruction 152 5.3.2 Manipulating elements 156 5.3.3 Testing and changing array size 162 5.4 HV functions 164 5.4.1 Creation and destruction 164 5.4.2 Manipulating elements 166 5.5 Miscellaneous functions 170 5.5.1 Memory management 170 5.5.2 Unicode data handling 175 5.5.3 Everything else 178 5.6 Summary 182 Advanced XS programming 183 6.1 Pointers and things 184 6.2 Filehandles 186 6.3 Typemaps 187 6.4 The argument stack 189 6.5 C structures 190 6.5.1 C structures as black boxes 190 6.5.2 C structures as objects 196 6.5.3 C structures as hashes 199 6.6 Arrays 203 6.6.1 Passing numeric arrays from Perl to C 203 6.6.2 Passing numeric arrays from C to Perl 210 6.6.3 The Perl Data Language 212 6.6.4 Benchmarks 218 6.6.5 Character strings 219 6.7 Callbacks 222 6.7.1 Immediate callbacks 223 6.7.2 Deferred callbacks 226 6.7.3 Multiple callbacks 227 6.8 Other languages 229 6.8.1 Linking Perl to C++ 229 6.8.2 Linking Perl to Fortran 236 6.8.3 Linking Perl to Java 243 6.9 Interface design: part 2 243 6.10 Older Perls 244 6.11 What’s really going on? 245 6.11.1 What does xsubpp generate? 246 6.12 Further reading 250 6.13 Summary 250 Alternatives to XS 251 7.1 The h2xs program 252 7.2 SWIG 253 7.2.1 Data structures 256 7.3 The Inline module 258 7.3.1 What is going on? 259 7.3.2 Additional Inline examples 260 7.3.3 Inline and CPAN 265 7.3.4 Inline module summary 266 7.4 The PDL::PP module 267 7.4.1 The .pd file 268 7.4.2 The Makefile.PL file 269 7.4.3 Pure PDL 271 7.5 Earlier alternatives 271 7.6 Further reading 272 7.7 Summary 273 Embedding Perl in C 274 8.1 When to embed 274 8.2 When not to embed 275 8.3 Things to think about 275 8.4 “Hello C” from Perl 275 8.4.1 Compiling embedded programs 277 8.5 Passing data 277 8.6 Calling Perl routines 279 8.6.1 Stack manipulation 281 8.6.2 Context 283 8.6.3 Trapping errors with eval 283 8.6.4 Calling Perl methods in C 284 8.6.5 Calling Perl statements 285 8.7 Using C in Perl in C 285 8.8 Embedding wisdom 286 8.9 Summary 287 Embedding case study 288 9.1 Goals 288 9.2 Preparing the ground 289 9.3 Configuration options 290 9.4 Testing options 293 9.4.1 Binary options 293 9.4.2 Quad-state options 294 9.4.3 String options 295 9.5 Summary 296 Introduction to Perl internals 297 10.1 The source tree 297 10.1.1 The Perl library 297 10.1.2 The XS library 298 10.1.3 The I/O subsystem 298 10.1.4 The Regexp engine 298 10.1.5 The parser and tokenizer 298 10.1.6 Variable handling 299 10.1.7 Runtime execution 299 10.2 The parser 299 10.2.1 BNF and parsing 299 10.2.2 Parse actions and token values 301 10.2.3 Parsing some Perl 301 10.3 The tokenizer 302 10.3.1 Basic tokenizing 302 10.3.2 Sublexing 304 10.3.3 Tokenizer summary 305 10.4 Op code trees 305 10.4.1 The basic op 305 10.4.2 The different operations 306 10.4.3 Different flavors of ops 306 10.4.4 Tying it all together 308 10.4.5 PP Code 310 10.4.6 The opcode table and opcodes.pl 313 10.4.7 Scratchpads and targets 313 10.4.8 The optimizer 314 10.4.9 Op code trees summary 314 10.5 Execution 315 10.6 The Perl compiler 315 10.6.1 What is the Perl compiler? 315 10.6.2 B:: modules 316 10.6.3 What B and O provide 319 10.6.4 Using B for simple tasks 320 10.7 Further reading 323 10.8 Summary 323 Hacking Perl 324 11.1 The development process 324 11.1.1 Perl versioning 324 11.1.2 The development tracks 325 11.1.3 The perl5-porters mailing list 325 11.1.4 Pumpkins and pumpkings 325 11.1.5 The Perl repository 326 11.2 Debugging aids 326 11.2.1 Debugging modules 327 11.2.2 The built-in debugger: perl -D 327 11.2.3 Debugging functions 330 11.2.4 External debuggers 330 11.3 Creating a patch 337 11.3.1 How to solve problems 337 11.3.2 Autogenerated files 338 11.3.3 The patch itself 339 11.3.4 Documentation 340 11.3.5 Testing 340 11.3.6 Submitting your patch 340 11.4 Perl 6: the future of Perl 341 11.4.1 A history 341 11.4.2 Design and implementation 342 11.4.3 What happens next 343 11.4.4 The future for Perl 5 343 11.5 Further reading 343 11.6 Summary 343 Appendix A: Perl’s typemaps 344 A.1 Quick refresher 344 A.2 The typemaps 345 A.2.1 T_SV 345 A.2.2 T_SVREF 346 A.2.3 T_AVREF 347 A.2.4 T_HVREF 347 A.2.5 T_CVREF 348 A.2.6 T_SYSRET 349 A.2.7 T_UV 350 A.2.8 T_IV 350 A.2.9 T_INT 351 A.2.10 T_ENUM 351 A.2.11 T_BOOL 352 A.2.12 T_U_INT 352 A.2.13 T_SHORT 352 A.2.14 T_U_SHORT 353 A.2.15 T_LONG 353 A.2.16 T_U_LONG 354 A.2.17 T_CHAR 354 A.2.18 T_U_CHAR 355 A.2.19 T_FLOAT 355 A.2.20 T_NV 356 A.2.21 T_DOUBLE 356 A.2.22 T_PV 357 A.2.23 T_PTR 357 A.2.24 T_PTRREF 358 A.2.25 T_PTROBJ 359 A.2.26 T_REF_IV_PTR 360 A.2.27 T_OPAQUEPTR 360 A.2.28 T_OPAQUE 362 A.2.29 T_PACKED 363 A.2.30 T_PACKEDARRAY 363 A.2.31 T_ARRAY 364 A.2.32 T_STDIO 366 A.2.33 T_INOUT 367 Appendix B: Further reading 368 Perl 368 C 369 Appendix C: Perl API index 370 index 375 contents......Page 7 preface......Page 13 acknowledgments......Page 15 about this book......Page 16 author online......Page 19 about the cover illustration......Page 20 1.1 Hello, world......Page 21 1.2 The C compiler......Page 22 1.3 Header files......Page 23 1.4 The main function......Page 24 1.5.1 Function parameters......Page 26 1.5.2 Automatic variables......Page 27 1.5.3 Global variables......Page 28 1.5.4 Static variables......Page 29 1.6 Data types......Page 30 1.6.1 C types......Page 31 1.6.2 Types defined in Perl......Page 35 1.7 Casting......Page 36 1.8.1 Statements and blocks......Page 37 1.8.2 The break and continue statements......Page 38 1.8.3 The switch statement......Page 39 1.9 Macros and the C preprocessor......Page 40 1.11 Summary......Page 43 2.1 Perl modules......Page 44 2.1.1 Module distributions......Page 46 2.2.1 The Perl module......Page 50 2.2.2 The XS file......Page 51 2.2.3 Example: “Hello, world”......Page 52 2.2.4 Return values......Page 56 2.2.5 Arguments and return values......Page 57 2.3.1 Modifying input variables......Page 58 2.3.2 Output arguments......Page 59 2.3.3 Compiler constants......Page 60 2.4 What about Makefile.PL?......Page 64 2.5 Interface design: part 1......Page 67 2.5.2 Don’t supply what is already known......Page 68 2.5.5 Use double precision......Page 69 2.7 Summary......Page 70 3.1 Arrays......Page 71 3.2 Pointers......Page 73 3.2.1 Pointers and arrays......Page 75 3.2.2 Pointers to functions......Page 77 3.3 Strings......Page 78 3.3.1 Arrays of strings......Page 79 3.4 Structures......Page 80 3.5 File I/O......Page 82 3.6 Memory management......Page 83 3.6.1 Allocating memory at runtime......Page 84 3.6.3 Manipulating memory......Page 85 3.6.4 Memory manipulation and Perl......Page 87 3.7 C Traps for the Perl programmer......Page 88 3.9 Summary......Page 89 4.1 General concepts......Page 90 4.1.2 Looking inside: Devel::Peek......Page 91 4.1.3 The flag system......Page 92 4.2.1 The SvNULL type......Page 94 4.2.3 SvPV: string values......Page 96 4.2.4 SvPVIV: integers......Page 98 4.2.5 SvPVNV: floating-point numbers......Page 99 4.2.7 SvOOK: offset strings......Page 100 4.3 Magic variables: SvPVMG......Page 101 4.4 Array variables......Page 105 4.5 Hashes......Page 107 4.6 Globs......Page 111 4.7 Namespaces and stashes......Page 114 4.8 Lexical “my” variables......Page 115 4.9 Code blocks......Page 116 4.9.1 Important CV flags......Page 117 4.11 Summary......Page 119 The Perl 5 API......Page 120 5.2.1 Special SVs......Page 121 5.2.2 Creating SVs......Page 123 5.2.3 Accessing data......Page 130 5.2.4 Manipulating data......Page 139 5.2.5 String functions......Page 144 5.2.6 References......Page 149 5.3.1 Creation and destruction......Page 152 5.3.2 Manipulating elements......Page 156 5.3.3 Testing and changing array size......Page 162 5.4.1 Creation and destruction......Page 164 5.4.2 Manipulating elements......Page 166 5.5.1 Memory management......Page 170 5.5.2 Unicode data handling......Page 175 5.5.3 Everything else......Page 178 5.6 Summary......Page 182 Advanced XS programming......Page 183 6.1 Pointers and things......Page 184 6.2 Filehandles......Page 186 6.3 Typemaps......Page 187 6.4 The argument stack......Page 189 6.5.1 C structures as black boxes......Page 190 6.5.2 C structures as objects......Page 196 6.5.3 C structures as hashes......Page 199 6.6.1 Passing numeric arrays from Perl to C......Page 203 6.6.2 Passing numeric arrays from C to Perl......Page 210 6.6.3 The Perl Data Language......Page 212 6.6.4 Benchmarks......Page 218 6.6.5 Character strings......Page 219 6.7 Callbacks......Page 222 6.7.1 Immediate callbacks......Page 223 6.7.2 Deferred callbacks......Page 226 6.7.3 Multiple callbacks......Page 227 6.8.1 Linking Perl to C++......Page 229 6.8.2 Linking Perl to Fortran......Page 236 6.9 Interface design: part 2......Page 243 6.10 Older Perls......Page 244 6.11 What’s really going on?......Page 245 6.11.1 What does xsubpp generate?......Page 246 6.13 Summary......Page 250 Alternatives to XS......Page 251 7.1 The h2xs program......Page 252 7.2 SWIG......Page 253 7.2.1 Data structures......Page 256 7.3 The Inline module......Page 258 7.3.1 What is going on?......Page 259 7.3.2 Additional Inline examples......Page 260 7.3.3 Inline and CPAN......Page 265 7.3.4 Inline module summary......Page 266 7.4 The PDL::PP module......Page 267 7.4.1 The .pd file......Page 268 7.4.2 The Makefile.PL file......Page 269 7.5 Earlier alternatives......Page 271 7.6 Further reading......Page 272 7.7 Summary......Page 273 8.1 When to embed......Page 274 8.4 “Hello C” from Perl......Page 275 8.5 Passing data......Page 277 8.6 Calling Perl routines......Page 279 8.6.1 Stack manipulation......Page 281 8.6.3 Trapping errors with eval......Page 283 8.6.4 Calling Perl methods in C......Page 284 8.7 Using C in Perl in C......Page 285 8.8 Embedding wisdom......Page 286 8.9 Summary......Page 287 9.1 Goals......Page 288 9.2 Preparing the ground......Page 289 9.3 Configuration options......Page 290 9.4.1 Binary options......Page 293 9.4.2 Quad-state options......Page 294 9.4.3 String options......Page 295 9.5 Summary......Page 296 10.1.1 The Perl library......Page 297 10.1.5 The parser and tokenizer......Page 298 10.2.1 BNF and parsing......Page 299 10.2.3 Parsing some Perl......Page 301 10.3.1 Basic tokenizing......Page 302 10.3.2 Sublexing......Page 304 10.4.1 The basic op......Page 305 10.4.3 Different flavors of ops......Page 306 10.4.4 Tying it all together......Page 308 10.4.5 PP Code......Page 310 10.4.7 Scratchpads and targets......Page 313 10.4.9 Op code trees summary......Page 314 10.6.1 What is the Perl compiler?......Page 315 10.6.2 B:: modules......Page 316 10.6.3 What B and O provide......Page 319 10.6.4 Using B for simple tasks......Page 320 10.8 Summary......Page 323 11.1.1 Perl versioning......Page 324 11.1.4 Pumpkins and pumpkings......Page 325 11.2 Debugging aids......Page 326 11.2.2 The built-in debugger: perl -D......Page 327 11.2.4 External debuggers......Page 330 11.3.1 How to solve problems......Page 337 11.3.2 Autogenerated files......Page 338 11.3.3 The patch itself......Page 339 11.3.6 Submitting your patch......Page 340 11.4.1 A history......Page 341 11.4.2 Design and implementation......Page 342 11.6 Summary......Page 343 A.1 Quick refresher......Page 344 A.2.1 T_SV......Page 345 A.2.2 T_SVREF......Page 346 A.2.4 T_HVREF......Page 347 A.2.5 T_CVREF......Page 348 A.2.6 T_SYSRET......Page 349 A.2.8 T_IV......Page 350 A.2.10 T_ENUM......Page 351 A.2.13 T_SHORT......Page 352 A.2.15 T_LONG......Page 353 A.2.17 T_CHAR......Page 354 A.2.19 T_FLOAT......Page 355 A.2.21 T_DOUBLE......Page 356 A.2.23 T_PTR......Page 357 A.2.24 T_PTRREF......Page 358 A.2.25 T_PTROBJ......Page 359 A.2.27 T_OPAQUEPTR......Page 360 A.2.28 T_OPAQUE......Page 362 A.2.30 T_PACKEDARRAY......Page 363 A.2.31 T_ARRAY......Page 364 A.2.32 T_STDIO......Page 366 A.2.33 T_INOUT......Page 367 Perl......Page 368 C......Page 369 Appendix C: Perl API index......Page 370 index......Page 375

Extending and Embedding Perl explains how to expand the functionality and usefulness of the Perl programming language and how to use Perl from C programs. It begins simply but also covers complex issues using real code examples from the Perl source. The book discusses how to write interfaces to C libraries (as well as C++ and Fortran libraries). It shows you how to implement Perl callbacks for C libraries, how to pass Perl hashes and arrays between Perl and C, and how to use the Perl Data Language infrastructure to improve the speed of array operations.

Additionally, the book peers under the hood to see how the Perl programming language really works by looking at the interpreter. The make-up of Perl variables is discussed along with details on how a Perl program is parsed and converted to executable code.

An explanation of how to expand the functionality and usefulness of the Perl programming language, this guide delves into the complex issues of using real code examples from the Perl source. Detailed is how to use Perl from C programs, such as writing interfaces to C libraries, techniques on implementing Perl callbacks for C libraries, passing Perl hashes and arrays between Perl and C. Additionally, developers are provided with an API reference for the internal C interface to Perl and a reference on the typemap system. An explanation of how to expand the functionality and usefulness of the Perl programming language, this guide delves into the complex issues of using real code examples from the Perl source. Detailed is how to use Perl from C programs, such as writing interfaces to C libraries, implementing Perl callbacks for C libraries, and passing Perl hashes and arrays between Perl and C. Additionally, developers are provided with an API reference for the internal C interface to Perl and a reference on the typemap system.