Building an event-driven data mesh : patterns for designing and building event-driven architectures

Building an Event-Driven Data MeshPatterns for Designing and Building Event-Driven ArchitecturesThe exponential growth of data combined with the need to derive real-time business value is a critical issue. An event-driven data mesh can power real-time operational and analytical workloads, all from a single set of data product streams. With practical real-world examples, this book provides patterns that show software architects and developers how to successfully design and build an event-driven data mesh.Author Adam Bellemare demonstrates what events and streams are, where they come from, and how you can use them. You’ll also examine design patterns, their implications, and trade-offs inherent in their use.This book provides:• A foundation for how events and event streams relate to the four pillars of data mesh• Practical tips for building an event-driven data mesh, including incremental integration with your existing systems• A clear understanding of how events relate to systems and other events, both in the same stream and across streams• A realistic look at event design options such as fact, delta, and command event types, including how these choices will impact your data products• Best practices for privacy, handling events at scale, and regulatory compliance• Advice on asynchronous communication and handling eventual consistency Cover Copyright Table of Contents Preface Conventions Used in This Book O’Reilly Online Learning How to Contact Us Acknowledgments Chapter 1. Event-Driven Data Communication What Is Data Mesh? An Event-Driven Data Mesh Using Data in the Operational Plane The Data Monolith The Difficulties of Communicating Data for Operational Concerns The Analytical Plane: Data Warehouses and Data Lakes The Organizational Impact of Schema on Read Bad Data: The Costs of Inaction Can We Unify Analytical and Operational Workflows? Rethinking Data with Data Mesh Common Objections to an Event-Driven Data Mesh Producers Cannot Model Data for Everyone’s Use Cases Making Multiple Copies of Data Is Bad Eventual Consistency Is Too Difficult to Manage Summary Chapter 2. Data Mesh Principle 1: Domain Ownership Domain-Driven Design in Brief Selecting the Data to Expose from Your Domain Principle 2: Data as a Product Data Products Provide Immutable and Time-Stamped Data Data Products Are Multimodal Accessing a Data Product Via Push or Pull The Three Data Product Alignment Types Event-Driven Data Products as Inputs for Operational Systems Principle 3: Federated Governance Specifying Data Product Language, Framework, and API Support Establishing Data Product Life Cycle Requirements Establishing Data Handling and Infosec Policies Identifying and Standardizing Cross-Domain Polysemes Formalizing Self-Service Platform Requirements Principle 4: Self-Service Platform Discovering Data Products and Dependencies Data Product Management Controls Data Product Access Controls Compute and Storage Resources for Building and Using Data Products Providing Self-Service Through SaaS Summary Chapter 3. Event Streams for Data Mesh Events, Messages, and Records What’s an Event Stream? What Is It Not? Ephemeral Message-Passing Queuing Consuming and Using Event-Driven Data Products State Events and Event-Carried State Transfer Materializing Events Aggregating Events The Kappa Architecture The Lambda Architecture and Why It Doesn’t Work for Data Mesh Supporting the Requirements for Kappa Architecture Selecting an Event Broker Summary Chapter 4. Federated Governance Forming a Federated Governance Team Implementing Standards Supporting Multimodal Data Product Types Supporting Data Product Schemas Supporting Programming Languages and Frameworks Metadata Standards and Requirements Ensuring Cross-Domain Data Product Compatibility and Interoperability Defining and Using Common Entities Event Stream Keying and Partitioning Time and Time Zones What Does a Governance Meeting Look Like? 1. Identifying Existing Problems 2. Drafting Proposals 3. Reviewing Proposals 4. Implementing Proposals 5. Archiving Proposals Data Security and Access Policies Disable Data Product Access by Default Consider End-to-End Encryption Field-Level Encryption Data Privacy, the Right to Be Forgotten, and Crypto-Shredding Data Product Lineage Topology-Based Lineage Record-Based Lineage Summary Chapter 5. Self-Service Data Platform The Self-Service Platform Maturity Model Level 1: The Minimal Viable Platform The Schema Registry An Extremely Basic Metadata Catalog Connectors Level 1 Wrap-Up: How Does It Work? Level 2: The Expanded Platform Full-Featured Metadata Catalog The Data Product Management Service and UI Service and User Identities Basic Access Controls Stream Processing for Building Data Products Level 2 Wrap-Up: How Does It Work? Level 3: The Mature Platform Authentication, Identification, and Access Management Integration with Existing Application Delivery Processes Programmatic Data Product Management API Monitoring and Alerting Multiregion and Multicloud Data Products Level 3 Wrap-Up: How Does It Work? Summary Chapter 6. Event Schemas A Brief Introduction to Serialization and Deserialization What Is a Schema? What Are Our Schema Technology Options? Google’s Protocol Buffers, aka Protobuf Apache Avro JSON Schema Schema Evolution: Changing Your Schemas Through Time Negotiating a Breaking Schema Change Step 1: Design the New Data Model Step 2: Iterate with Your Existing Consumers and the Federated Governance Team Step 3. Create a Release Schedule, a Data Migration Plan, and a Deprecation Plan Step 4. Execute the Release The Role of the Schema Registry Best Practices for Managing Schemas in Your Codebase Choosing a Schema Technology Summary Chapter 7. Designing Events Introduction to Event Types Expanding on State Events and Event-Carried State Transfer Current State Events Before/After State Events Delta Events Event Sourcing with Delta Events Why Delta Events Don’t Work for Event-Driven Data Products Measurement Events Measurement Events Often Form Aggregate-Aligned Data Products Measurement Event Sources May Be Lossy Measurement Events May Power Time-Sensitive Applications Hybrid Events—State with a Bit of Delta Notification Events Summary Chapter 8. Bootstrapping Data Products Getting Started: Bootstrapping with Connectors Dual Writes Polling the Database to Create Data Products Change-Data Capture Change-Data Capture Using a Transactional Outbox Denormalization and Eventification Eventification at the Transactional Outbox Eventification in a Dedicated Service What Should Go In the Event? And What Should Stay Out? Slowly Changing Dimensions Bootstrapping Cloud Storage Files to an Event Stream Summary Chapter 9. Integrating Event-Driven Data into Data at Rest Analytics and the Medallion Architecture Connecting Event Streams Into Existing Batch-Data Flows Through the Lens of Data Mesh: What’s Going On? Through the Lens of Data Mesh: How Do We Solve It? Balancing File Sizes, SLAs, and Latency Budget Blues: A Tale of Overspending Extending the Self-Service Platform for Nonstreaming Data Products Summary Chapter 10. Eventual Consistency Converging on Consistency, One Event at a Time Strategies for Dealing with Eventual Consistency Prevent Failures to Avoid Inconsistency Use Event-Driven Data Products Instead of Request-Response Server API Calls Expose Eventual Consistency in the Server Response Plan for New Services and Reprocessing of Data Synchronize Data Products on Time Boundaries Out-of-Order Events Resolving Late-Arriving Events Summary Chapter 11. Bringing It All Together Event Streams for Data Mesh Integrating with Existing Systems Operations, Analytics, and Everything in Between Summary Index About the Author Colophon The exponential growth of data combined with the need to derive real-time business value is a critical issue today. An event-driven data mesh can power real-time operational and analytical workloads, all from a single set of data product streams. With practical real-world examples, this book shows you how to successfully design and build an event-driven data mesh.Building an Event-Driven Data Mesh provides:Practical tips for iteratively building your own event-driven data mesh, including hurdles you'll experience, possible solutions, and how to obtain real value as soon as possibleSolutions to pitfalls you may encounter when moving your organization from monoliths to event-driven architecturesA clear understanding of how events relate to systems and other events in the same stream and across streamsA realistic look at event modeling options, such as fact, delta, and command type events, including how these choices will impact your data productsBest practices for handling events at scale, privacy, and regulatory complianceAdvice on asynchronous communication and handling eventual consistency