Kafka Streams in Action, Second Edition - MEAP Version 8

Kafka Streams in Action, Second Edition MEAP V08 Copyright Welcome Brief Contents Chapter 1: Welcome to the kafka event streaming platform 1.1 What is event streaming ? 1.1.1 What is an event ? 1.1.2 An event stream example 1.1.3 Who needs event streaming applications 1.2 Introducing the Apache Kafka® event streaming platform 1.2.1 Kafka brokers 1.2.2 Schema registry 1.2.3 Producer and consumer clients 1.2.4 Kafka Connect 1.2.5 Kafka Streams 1.2.6 ksqlDB 1.3 A concrete example of applying the Kafka event streaming platform 1.4 Summary Chapter 2: Kafka brokers 2.1 Produce record requests 2.2 Consume record requests 2.3 Topics and partitions 2.3.1 Offsets 2.3.2 Determining the correct number of partitions 2.4 Sending your first messages 2.4.1 Creating a topic 2.4.2 Producing records on the command line 2.4.3 Consuming records from the command line 2.4.4 Partitions in action 2.5 Segments 2.5.1 Data retention 2.5.2 Compacted topics 2.5.3 Topic partition directory contents 2.6 Tiered storage 2.7 Cluster Metadata 2.8 Leaders and followers 2.8.1 Replication 2.9 Checking for a healthy broker 2.9.1 Request handler idle percentage 2.9.2 Network handler idle percentage 2.9.3 Under replicated partitions 2.10 Summary Chapter 3: Schema registry 3.1 What is a schema and why you need to use one 3.1.1 What is Schema Registry? 3.1.2 Getting Schema Registry 3.1.3 Architecture 3.1.4 Communication - Using Schema Registry’s REST API 3.1.5 Plugins and serialization platform tools 3.2 Subject name strategies 3.2.1 TopicNameStrategy 3.2.2 RecordNameStrategy 3.2.3 TopicRecordNameStrategy 3.3 Schema compatibility 3.3.1 Backward compatibility 3.3.2 Forward compatibility 3.3.3 Full compatibility 3.3.4 No compatibility 3.4 Schema references 3.5 Schema references and multiple events per topic 3.6 Schema Registry (de)serializers 3.6.1 Avro 3.6.2 Protobuf 3.6.3 JSON Schema 3.7 Serialization without Schema Registry 3.8 Summary Chapter 4: Kafka clients 4.1 Producing records with the KafkaProducer 4.1.1 Producer configurations 4.1.2 Kafka delivery semantics 4.1.3 Partition assignment 4.1.4 Writing a custom partitioner 4.1.5 Specifying a custom partitioner 4.1.6 Timestamps 4.2 Consuming records with the KafkaConsumer 4.2.1 The poll interval 4.2.2 Group id 4.2.3 Static membership 4.2.4 Committing offsets 4.3 Exactly once delivery in Kafka 4.3.1 Idempotent producer 4.3.2 Transactional producer 4.3.3 Consumers in transactions 4.3.4 Producers and consumers within a transaction 4.4 Using the Admin API for programmatic topic management 4.4.1 Working with topics programmatically 4.5 Handling multiple event types in a single topic 4.5.1 Producing multiple event types 4.5.2 Consuming multiple event types 4.6 Summary Chapter 5: Kafka connect 5.1 Integrating external applications into Kafka 5.2 Getting Started with Kafka Connect 5.3 Applying Single Message Transforms 5.3.1 Adding a Sink Connector 5.4 Building and deploying your own Connector 5.4.1 Implementing a connctor 5.4.2 Making your connector dynamic with a monitoring thread 5.4.3 Creatign a custom transformation 5.5 Summary Chapter 6: Developing Kafka Streams 6.1 The Streams DSL 6.2 Hello World for Kafka Streams 6.2.1 Creating the topology for the Yelling App 6.2.2 Kafka Streams configuration 6.2.3 Serde creation 6.3 Masking credit card numbers and tracking purchase rewards in a retail sales setting 6.3.1 Building the source node and the masking processor 6.3.2 Adding the patterns processor 6.3.3 Building the rewards processor 6.3.4 Using Serdes to encpsulate serializers and deserializers in Kafka Streams 6.3.5 Kafka Streams and Schema Registry 6.4 Interactive development 6.5 Choosing which events to process 6.5.1 Filtering purchases 6.5.2 Splitting/branching the stream 6.5.3 Naming topology nodes 6.5.4 Dynamic routing of messages 6.6 Summary Chapter 7: Streams and state 7.1 Stateful vs stateless 7.2 Adding stateful operations to Kafka Streams 7.2.1 Group By details 7.2.2 Aggregation vs. reducing 7.2.3 Repartitioning the data 7.2.4 Proactive Repartitioning 7.2.5 Repartitioning to increase the number of tasks 7.2.6 Using Kafka Streams Optimizations 7.3 Stream-Stream Joins 7.3.1 Implementing a stream-stream join 7.3.2 Join internals 7.3.3 ValueJoiner 7.3.4 Join Windows 7.3.5 StreamJoined 7.3.6 Other join options 7.3.7 Outer joins 7.3.8 Left-outer join 7.4 State stores in Kafka Streams 7.4.1 Changelog topics restoring state stores 7.4.2 Standby Tasks 7.4.3 Assigning state stores in Kafka Streams 7.4.4 State store location on the file system 7.4.5 Naming Stateful operations 7.4.6 Specifying a store type 7.4.7 Configuring changelog topics 7.5 Summary Chapter 8: Advanced stateful concepts 8.1 KTable The Update Stream 8.1.1 Updates to records or the changelog 8.1.2 Event streams vs. update streams 8.2 KTables are stateful 8.3 The KTable API 8.4 KTable Aggregations 8.5 GlobalKTable 8.6 KTable Joins 8.7 Stream-Table join details 8.8 Table-Table join details 8.9 Stream-GlobaTable join details 8.10 Windowing 8.11 Out order records and grace 8.12 Tumbling windows 8.13 Session windows 8.14 Sliding windows 8.15 Suppression 8.16 Timestamps in Kafka Streams 8.17 The TimestampExtractor 8.18 WallclockTimestampExtractor 8.19 Custom TimestampExtractor 8.20 Specifying a TimestampExtractor 8.21 Streamtime 8.22 Summary Chapter 9: The Processor API 9.1 The trade-offs of higher-level abstractions vs. more control 9.2 Working with sources, processors, and sinks to create a topology 9.2.1 Adding a source node 9.2.2 Adding a processor node 9.2.3 Adding a sink node 9.3 Digging deeper into the Processor API with a stock analysis processor 9.3.1 The stock-performance processor application 9.3.2 The process() method 9.3.3 The punctuator execution 9.4 Data Driven Aggregation 9.5 Integrating the Processor API and the Kafka Streams API 9.6 Summary Appendix B: Schema compatibility workshop B.1 Backward compatibility B.2 Forward compatibility B.3 Full compatibility Notes Everything you need to implement stream processing on Apache Kafka using Kafka Streams and the kqsIDB event streaming database.Kafka Streams in Action, Second Edition guides you through setting up and maintaining your streaming processing with Kafka. Inside, you'll find comprehensive coverage of not only Kafka Streams, but the entire toolbox you'll need for effective streaming—from the components of the Kafka ecosystem, to Producer and Consumer clients, Connect, and Schema Registry. In Kafka Streams in Action, Second Edition you'll learn how to: Design streaming applications in Kafka Streams with the KStream and the Processor API Integrate external systems with Kafka Connect Enforce data compatibility with Schema Registry Build applications that respond immediately to events in either Kafka Streams or ksqlDB Craft materialized views over streams with ksqlDB This totally revised new edition of Kafka Streams in Action has been expanded to cover more of the Kafka platform used for building event-based applications. You'll also find full coverage of ksqlDB, an event streaming database that makes it a snap to create applications that respond immediately to events, such as real-time push and pull updates. Foreword by Jun Rao. About the technology Enterprise applications need to handle thousands—even millions—of data events every day. With an intuitive API and flawless reliability, the lightweight Kafka Streams library has earned a spot at the center of these systems. Kafka Streams provides exactly the power and simplicity you need to manage real-time event processing or microservices messaging. About the book Kafka Streams in Action, Second Edition teaches you how to create event streaming applications on the amazing Apache Kafka platform. This thoroughly revised new edition now covers a wider range of streaming architectures and includes data integration with Kafka Connect. As you go, you'll explore real-world examples that introduce components and brokers, schema management, and the other essentials. Along the way, you'll pick up practical techniques for blending Kafka with Spring, low-level control of processors and state stores, storing event data with ksqlDB, and testing streaming applications. What's inside Design efficient streaming applications Integrate external systems with Kafka Connect Enforce data compatibility with Schema Registry About the reader For Java developers. No knowledge of Kafka or streaming applications required. About the author Bill Bejeck is a Confluent engineer and a Kafka Streams contributor with over 15 years of software development experience. Bill is also a committer on the Apache Kafka? project. Table of Contents PART 1 1 Welcome to the Kafka event streaming platform 2 Kafka brokers PART 2 3 Schema Registry 4 Kafka clients 5 Kafka ConnectPART 3 6 Developing Kafka Streams 7 Streams and state 8 The KTable API 9 Windowing and timestamps 10 The Processor API 11 ksqlDB 12 Spring kafka 13 Kafka Streams Interactive Queries 14 Testing