Pro Spring Batch : batch processing with the Spring Batch framework

Since its release, Spring Framework has transformed virtually every aspect of Java development including web applications, security, aspect-oriented programming, persistence, and messaging. Spring Batch, one of its newer additions, now brings the same familiar Spring idioms to batch processing. Spring Batch addresses the needs of any batch process, from the complex calculations performed in the biggest financial institutions to simple data migrations that occur with many software development projects. Pro Spring Batch is intended to answer three questions: What? What is batch processing? What does it entail? What makes it different from the other applications we are developing? What are the challenges inherent in the development of a batch process? Why? Why do batch processing? Why can’t we just process things as we get them? Why do we do batch processing differently than the web applications that we currently work on? How? How to implement a robust, scalable, distributed batch processing system using open-source frameworks Pro Spring Batch gives concrete examples of how each piece of functionality is used and why it would be used in a real-world application. This includes providing tips that the "school of hard knocks" has taught author Michael Minella during his experience with Spring Batch. Pro Spring Batch includes examples of I/O options that are not mentioned in the official user’s guide, as well as performance tips on things like how to limit the impact of maintaining the state of your jobs. The author also walks you through, from end to end, the design and implementation of a batch process based upon a theoretical real-world example. This includes basic project setup, implementation, testing, tuning and scaling for large volumes. What you’ll learn Batch concepts and how they relate to the Spring Batch framework How to use declarative I/O using the Spring Batch readers/writers Data integrity techniques used by Spring Batch, including transactions and job state/restartability How to scale batch jobs via distributed batch processing How to handle testing batch processes (Unit and functional) Who this book is for Java developers with Spring experience. Java Architects designing batch solutions More specifically, this book is intended for those who have a solid foundation in the core Java platform. Batch processing covers a wide spectrum of topics, not all of which are covered in detail in this book. Concepts in Java which the reader should be comfortable with include file I/O, JDBC, and transactions. Given that Spring Batch is a framework built upon the open-source IoC container Spring, which will not be covered in this book, it is expected that the reader will be familiar with its concepts and conventions. With that in mind, the reader is not expected to have any prior exposure to the Spring Batch framework. All concepts related to it will be explained in detail, with working examples. Table of Contents Batch and Spring Spring Batch 101 Sample Job Understanding Jobs and Steps Job Repository and Metadata Running a Job Readers Item Processors Item Writers Sample Application Scaling and Tuning Testing Batch Processes Cover......Page 1 Contents at a Glance......Page 3 Contents......Page 489 About the Author......Page 496 About the Technical Reviewer......Page 497 Acknowledgments......Page 498 Batch and Spring......Page 4 A History of Batch Processing......Page 5 Batch Challenges......Page 6 Why Do Batch Processing in Java?......Page 7 Other Uses for Spring Batch......Page 9 The Spring Batch Framework......Page 10 Managing Jobs......Page 11 The Spring Batch Admin Project......Page 12 Summary......Page 13 The Architecture of Batch......Page 14 Examining Jobs and Steps......Page 15 Job Execution......Page 16 Parallelization......Page 17 Batch Administration......Page 19 Documentation......Page 20 Obtaining Spring Batch......Page 22 It’s the Law: Hello, World!......Page 23 Running Your Job......Page 25 Job Repository Configuration......Page 26 The Job Repository Tables......Page 27 Summary......Page 31 Understanding Agile Development......Page 32 Capturing Requirements with User Stories......Page 33 Working with a True Development Environment......Page 35 Understanding the Requirements of the Statement Job......Page 36 Designing a Batch Job......Page 42 Job Description......Page 43 Understanding the Data Model......Page 45 Summary......Page 46 Introducing a Job......Page 47 Tracing a Job’s Lifecycle......Page 49 Configuring a Job......Page 50 Basic Job Configuration......Page 51 Job Inheritance......Page 53 Job Parameters......Page 54 Working with Job Listeners......Page 61 ExecutionContext......Page 63 Manipulating the ExecutionContext......Page 64 Chunk vs. Item Processing......Page 67 Step Configuration......Page 68 Understanding the Other Types of Tasklets......Page 70 Step Flow......Page 85 Item Error Handling......Page 95 Summary......Page 100 Using an In-Memory Job Repository......Page 101 Database......Page 103 The JobExplorer......Page 106 The JobOperator......Page 110 Summary......Page 120 Job Execution......Page 121 Spring Batch Job Runners......Page 122 Third-Party Integration......Page 126 Programmatic Ending......Page 144 External Stoppage......Page 157 Error Handling......Page 159 Preventing a Job from Being Rerun......Page 162 Configuring the Number of Restarts......Page 163 Rerunning a Complete Step......Page 164 Summary......Page 165 The ItemReader Interface......Page 166 Flat Files......Page 167 XML......Page 199 Database Input......Page 202 JDBC......Page 203 Hibernate......Page 209 JPA......Page 215 Existing Services......Page 219 Custom Input......Page 222 Skipping Records......Page 227 Logging Invalid Records......Page 229 Dealing with No Input......Page 231 Summary......Page 232 Introduction to ItemProcessors......Page 233 ValidatingItemProcessor......Page 235 ItemProcessorAdapter......Page 243 CompositeItemProcessor......Page 248 Writing Your Own ItemProcessor......Page 256 Filtering Items......Page 257 Summary......Page 258 Item Writers......Page 259 Introduction to ItemWriters......Page 260 FlatFileItemWriter......Page 261 StaxEventItemWriter......Page 273 JdbcBatchItemWriter......Page 278 HibernateItemWriter......Page 285 JpaItemWriter......Page 290 ItemWriterAdapter......Page 294 PropertyExtractingDelegatingItemWriter......Page 296 JmsItemWriter......Page 298 SimpleMailMessageItemWriter......Page 302 Multipart ItemWriters......Page 310 MultiResourceItemWriter......Page 311 CompositeItemWriter......Page 322 ClassifierCompositeItemWriter......Page 326 Summary......Page 330 Reviewing the Statement Job......Page 331 Setting Up a New Project......Page 332 Importing Customer and Transaction Data......Page 336 Creating the Customer Transaction Reader......Page 338 Looking Ip Ids......Page 346 Writing the Customer and Transaction Data......Page 353 Downloading Current Stock Prices......Page 355 Reading the Tickers......Page 356 Writing the Stock File......Page 360 Importing Current Stock Prices......Page 361 Reading the Stock Price File......Page 362 Writing Stock Prices to the Database......Page 363 Calculating Pricing Tiers......Page 364 Reading How Many Transactions the Customer Had......Page 365 Calculating the Pricing Tier......Page 367 Updating the Database with the Calculated Tier......Page 369 Reading the Transactions......Page 370 Saving Transaction Fees to the Database......Page 374 Reading the Statement Data......Page 375 Writing the Statements......Page 381 Summary......Page 385 Profiling Your Batch Process......Page 386 A Tour of VisualVM......Page 387 Profiling Spring Batch Applications......Page 391 Multithreaded Steps......Page 398 Parallel Steps......Page 403 Remote Chunking......Page 420 Partitioning......Page 434 Summary......Page 444 Unit Tests with JUnit and Mockito......Page 445 JUnit......Page 446 Mock Objects......Page 449 Mockito......Page 450 General Integration Testing with Spring......Page 454 Testing Spring Batch......Page 459 Summary......Page 468 B......Page 469 C......Page 470 E......Page 471 G......Page 472 I......Page 473 J......Page 474 M......Page 476 O......Page 477 P......Page 479 R......Page 480 S......Page 481 T......Page 483 V......Page 484 Z......Page 485