Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh

As data continues to grow and become more complex, organizations seek innovative solutions to manage their data effectively. Data mesh is one solution that provides a new approach to managing data in complex organizations. This practical guide offers step-by-step guidance on how to implement data mesh in your organization. In this book, Jean-Georges Perrin and Eric Broda focus on the key components of data mesh and provide practical advice supported by code. Data engineers, architects, and analysts will explore a simple and intuitive process for identifying key data mesh components and data products. You'll learn a consistent set of interfaces and access methods that make data products easy to consume. This approach ensures that your data products are easily accessible and the data mesh ecosystem is easy to navigate. This book helps you: • Identify, define, and build data products that interoperate within an enterprise data mesh • Build a data mesh fabric that binds data products together • Build and deploy data products in a data mesh • Establish the organizational structure to operate data products, data platforms, and data fabric • Learn an innovative architecture that brings data products and data fabric together into the data mesh About the authors: Jean-Georges "JG" Perrin is a technology leader focusing on building innovative and modern data platforms. Eric Broda is a technology executive, practitioner, and founder of a boutique consulting firm that helps global enterprises realize value from data. Copyright Table of Contents Foreword Preface Who Is This Book For? Overview of the Parts and Chapters What This Book Isn’t Conventions Used in This Book O’Reilly Online Learning How to Contact Us Acknowledgments Part I. The Basics Chapter 1. Understanding Data Mesh: The Essentials Making Data Agile Local Autonomy + Speed = Agility Solving Today’s Data Challenges Bridging Data Silos Shifting Toward Higher Quality Data Transforming Data Governance Data Volume, Variety, and Variability Turning Principles into Practice Summary Chapter 2. Applying Data Mesh Principles Data Mesh Principles Data as a Product Decentralized Domain Ownership Self-Serve Data Platform Federated Computational Governance Defining a “Good” Data Product Defining a Principled Data Product Defining a FAIR Data Product Defining an Enterprise-Grade Data Product Defining a Valuable Data Product Defining a Balanced Data Product Defining a Modern Data Product—More than Just “Regular” Data Defining a Practical Data Product Lifecycle Defining a Practical Data Mesh Roadmap A “Good” Data Product Has an Empowered Data Product Owner Identifying a Data Product Summary Chapter 3. Our Case Study: Climate Quantum Inc. Making Climate Data Easier to Find, Consume, Share, and Trust Introducing Climate Quantum Inc. Climate Quantum Data Landscape Applying Climate Quantum Inc. to Your Enterprise Summary Part II. Designing, Building, and Deploying Data Mesh Chapter 4. Defining the Data Mesh Architecture Data Product Architecture Data Product Artifacts Development Architecture Components Runtime Architecture Components Operations Architecture Components Data Mesh Architecture Data Mesh Marketplace and Registry Data Mesh Backbone Services Climate Quantum Use Case Considerations Summary Chapter 5. Driving Data Products with Data Contracts Bringing Value Through Trust Navigating the Data Contract Going Through the Theory Stacking Up Good Information It’s All About Proper Versioning Keeping It Simple and Semantic Walking Through an Example: Complementing Tribal Knowledge What is Data QoS and Why Is It Critical? Representation Why Does It Matter? Why Data Quality Is Not Enough Service Levels Complement Quality Applying Data QoS to the Data Contract Checking Conformity of Measurements Completeness Accuracy Engaging Service Levels Summary Chapter 6. Building Your First Data Product Anatomy of a Data Product The Data Contract to the Rescue Connecting Your Data Sources Ensuring Higher Data Quality with Observability Getting Faster Data Discovery Enabling Operability and Control Building and Packaging Summary Chapter 7. Aligning with the Experience Planes The Three Planes The Infrastructure Plane Remains Key The Data Product Experience Plane Is Independent The Mesh Experience Plane Is About Synergy Building a Capability Model for Each Plane Capabilities of the Infrastructure Experience Plane Capabilities of the Data Product Experience Plane Capabilities of the Mesh Experience Plane Gathering Key Metrics Through Feedback Loops Summary Chapter 8. Meshing Your Data Products Registering Your Data Product Connecting to Your Products in the Mesh Meshing Data Products Together Describing Your Data Lineage Notification Through the Mesh Summary Part III. GenAI, Teams, Operating Model, and Roadmap for Data Mesh Chapter 9. Running and Operating Your Data Mesh Making Data Products Discoverable, Observable, and Secure Data Product Discovery Interface Data Product Observability Interface Data Product Control Interface Data Product Security Climate Quantum Use Case Considerations Toward Dynamic Data Products The Power of Dynamic Data Products Summary Chapter 10. Creating a Data Mesh Marketplace Challenges with Traditional Data Catalogs The Data Mesh Marketplace A Window into Data Mesh The Consumer User Journey The Producer User Journey Self-Publishing of Data Products The Power of Discoverability Bootstrapping Your Marketplace Climate Quantum’s Data Mesh Marketplace Summary Chapter 11. Establishing Data Mesh Governance Data Governance Data Product Certification Federated Data Product Certification Implementing Data Product Certification Data Contracts and Certification Climate Quantum’s Certification Approach Summary Chapter 12. Understanding Data Product Supply Chains The Modern Manufacturing Supply Chain The Modern Software Factory The Data Product Factory The Data Product Supply Chain Climate Quantum’s Data Product Supply Chain Summary Chapter 13. Integrating Data Mesh and Generative AI Generative AI Background Large Language Models Embeddings Vector Databases Prompts Challenges Data Mesh and Generative AI Improve Data Quality with Generative AI Elevate Models as Primary Artifacts Consume Models Inside Data Products Generate Data Product Code Complement Data Product Capabilities Climate Quantum and Generative AI Simplifying Climate Data Search Analyzing and Summarizing Climate Data Creating Disclosure Reports Generating Code to Simplify Climate Data Processing Summary Chapter 14. Establishing Data Mesh Teams Team Topologies in Data Mesh The Data Product Team Platform Teams Enabling Teams Climate Quantum Teams Summary Chapter 15. Defining a Data Mesh Operating Model Characteristics of an Operating Model The Operating Model Continuum Centralized Organizations Matrixed Organizations Federated Organizations Distributed Organizations Data Mesh Operating Model Data Mesh Maturity Model Climate Quantum’s Data Mesh Operating Model Summary Chapter 16. Establishing a Practical Data Mesh Roadmap Roadmap Structure Strategy and Roadmap Stream Technology Stream Factory Stream Operating Model Stream Socialization Stream Rollout Stream Climate Quantum’s Data Mesh Roadmap Summary Index About the Authors Colophon