Data Mesh in Action - MEAP Version 4

Revolutionize the way your organization approaches data with a data mesh! This new decentralized architecture outpaces monolithic lakes and warehouses and can work for a company of any size. In Data Mesh in Action you will learn how to: * Implement a data mesh in your organization * Turn data into a data product * Move from your current data architecture to a data mesh * Identify data domains, and decompose an organization into smaller, manageable domains * Set up the central governance and local governance levels over data * Balance responsibilities between the two levels of governance * Establish a platform that allows efficient connection of distributed data products and automated governance Data Mesh in Action reveals how this groundbreaking architecture looks for both small startups and large enterprises. Youll see a data mesh in action as you explore both an extended case study and multiple real-world examples. As you go, youll be expertly guided through discussions around Socio-Technical Architecture and Domain-Driven Design with the goal of building a sleek data-as-a-product system. Data Mesh in Action MEAP V04 Copyright Welcome brief contents Front matter preface acknowledgments about this book Who should read this book How to Navigate this Book How to Use this Book The Messflix Case Study About the authors Chapter 1: The What and Why of Data Mesh 1.1 Data Mesh 101 Definition of a Data Mesh 1.2 Why the Data Mesh? 1.2.1 Alternatives 1.2.2 Data Warehouses & Data Lakes Inside the Data Mesh 1.2.3 Data Mesh benefits The business perspective The technology perspective 1.3 Use Case: The Snow Shoveling Business 1.4 Data Mesh principles 1.4.1 Domain-oriented decentralized data ownership and architecture 1.4.2 Data as a product When can we call data a product? Data Product as an autonomous component 1.4.3 Federated computational governance Federalization of Data Governance Computational elements of Data Governance 1.4.4 Self-serve data infrastructure as a platform 1.5 Back to Snow Shoveling 1.6 Socio-technical architecture 1.6.1 Conway’s law 1.6.2 Team Topologies 1.6.3 Cognitive Load 1.7 Data Mesh challenges 1.7.1 Technological challenge 1.7.2 Data Management challenge 1.7.3 Organizational challenge 1.8 Summary Chapter 2: Is Data Mesh Right for You? 2.1 Analyzing Data Mesh drivers 2.1.1 Business drivers Business strategy Business case and its data needs complexity 2.1.2 Organizational drivers Socio-technical complexity Data maturity Software engineering maturity (CI/CD, DevOps) 2.1.3 Domain-data drivers Domain and data model complexity Data diversity Data volume 2.1.4 Minor organizational drivers Data governance maturity Data-savvy software engineers Following Domain-Driven Design approach 2.1.5 Is Data Mesh a good fit for me? 2.2 Data Mesh alternatives and complementary solutions 2.2.1 Enterprise Data Warehouse 2.2.2 Data Lake 2.2.3 Data Lakehouse 2.2.4 Data Fabric 2.2.5 Data Mesh vs. the rest of the world 2.3 Understanding a Data Mesh implementation effort 2.3.1 Data Mesh development cycle 2.3.2 Development cycle in the shoveling example 2.3.3 Enabling the team Ensuring data governance Facilitating Data Product development Facilitating platform development Facilitating governance 2.3.4 Development cycle in detail Choose a business goal to enable with data Define the Data Products needed Develop Data Products Collect feedback about the platform and analyze common needs Establish common policies and improve governance Develop the platform 2.4 Summary Chapter 3: Kickstart your Data Mesh MVP in a month 3.1 Getting the lay of the land 3.1.1 Drawing a system landscape diagram 3.1.2 Stakeholder analysis 3.2 Identifying candidates for the MVP implementation team 3.2.1 Choosing development teams Choosing the cooperation model 3.2.2 Choosing a data governance team Why do you need a data governance team? Who do you need in a data governance team? How do you select a data governance team? 3.3 Setting up the minimal governance 3.3.1 Defining Data Mesh value statement(s) 3.3.2 Defining data governance policies 3.3.3 Federating data governance 3.4 Developing minimal Data Products 3.4.1 Identifying domain-oriented datasets Adding business benefit to the mix 3.4.2 Choosing Data Product Owners 3.4.3 Deciding on the minimum viable data product description 3.4.4 Developing the simplest tools to expose your data 3.5 Setting up the minimal platform 3.5.1 Ensuring platform-forced governability 3.5.2 Ensuring platform security 3.6 Summary Chapter 4: Domain ownership 4.1 Capturing and analyzing domains 4.1.1 Domain-Driven Design 101 4.1.2 Invite the right people 4.1.3 Choose the correct workshop technique Domain Storytelling 4.2 Apply ownership using domain decomposition 4.2.1 Domain, subdomain, and business capability 4.2.2 Decompose domain using business capability modeling 4.2.3 How are domains and business capabilities related to data? Domain, business capability, Domain Dataset, and Data Products Application to case study 4.2.4 Assign responsibilities to the Data Product-owning team Data Data pipelines, software, and source code Quality, cleansing, and deduplication Enriching and aggregations 4.2.5 Choose the right team to own data 4.3 Apply ownership using data use cases 4.3.1 Data use cases Use case description template 4.3.2 Model and bounded context 4.3.3 Set up boundaries of use case-driven Data Products Financial Analysis/Reporting Data Product Script recommendations Data Product Movie trends Data Product 4.3.4 Choose the right team to own data 4.4 Apply ownership using design heuristics 4.4.1 What is heuristic? 4.4.2 Using design heuristics 4.4.3 Designing heuristics and possible boundaries Align Data Product with a domain/business capability Align Data Products with possible usage or use cases Align Data Products with personas of consumers Align Data Product with the source system Align Data Product with the consuming dashboard/visualization/business intelligence tool Build the Data Product as a registry for a core business entity Create Data Product for a coherent group of Domain Datasets Avoid Data Products with a canonical model of the whole enterprise Align with usage contract Align with organizational boundaries 4.5 Final landscape: The mesh of interconnected Data Products 4.5.1 Messflix Data Mesh 4.5.2 Data Products form a mesh 4.5.3 Is it already a Data Mesh? 4.6 Summary Chapter 5: Data as a Product 5.1 Applying Product Thinking 5.1.1 Product Thinking analysis Cost Statement Data Product Scripts Data Product Movie Popularity Data Product 5.1.2 Data Product Canvas Movie Cast Data Product Canvas Movie Trends Data Product Canvas 5.2 What is a Data Product? 5.2.1 Data Product definition 5.2.2 Product not project 5.2.3 What can be a Data Product? 5.3 Data Product ownership 5.3.1 Data Product Owner 5.3.2 Data Product Owner responsibilities 5.3.3 An agile DevOps team as a base for Data Product Development Team 5.3.4 Data Product Owner and Product Owner 5.4 Conceptual architecture of a Data Product 5.4.1 External architecture view Data Product ports Database-like storage Files (REST) API Streams Visualisations 5.4.2 Internal architecture view Datasets Metadata Code 5.5 Data Product fundamental characteristics 5.5.1 Self-described Data Product 5.5.2 Introduction to Metadata 5.5.3 Metadata as code 5.5.4 Data Product metadata 5.5.5 Domain Data Set metadata 5.5.6 Other kinds of metadata 5.6 Additional Data Product characteristics: FAIR and immutability 5.6.1 Findability 5.6.2 Accessibility 5.6.3 Interoperable 5.6.4 Reusable 5.6.5 Immutable Do Data Products always have to be immutable? 5.7 Summary Chapter 6: Federated Computational Governance 6.1 Data Governance in a nutshell 6.2 Benefits of Data Governance 6.2.1 Business value perspective 6.2.2 Data usability perspective 6.2.3 Data control perspective 6.3 Planning Data Governance outcomes 6.3.1 Hierarchy of Data Governance outcomes 6.3.2 Strategic level outcomes Defining Value Statements Data policies Technology stack-related policies 6.3.3 Tactical level outcomes Detailing policies Assigning accountability and responsibility for data 6.3.4 Implementation-level outcomes Domain governance Central platform governance 6.4 Federating Data Governance Thinking of Data Governance in terms of “sliders” 6.4.1 Extreme ends of data governance models Centralized Data Governance Model: Top-down decision flow Decentralized Data Governance Model: Bottom-up decision flow 6.4.2 Federated Data Governance model Decision space focus Governance structure focus People focus Messflix Data Governance Council Messflix Data Governance Steering Committee Messflix Central Platform Team Messflix Data Product Owner 6.4.3 Setting-up Governance Team operations 6.5 Making Data Governance computational 6.5.1 Making policies Computational 6.5.2 Automating policy checks 6.6 Summary Chapter 7: The self-serve data platform 7.1 The MVP platform 7.1.1 Platform definition Application to our platform 7.1.2 Platform thinking Application to our platform 7.2 Improvements with the concept ”X-as-a-Service” 7.2.1 X-as-a-Service explained Workshop Tool-Building-as-a-Service X-as-a-Service continued X-as-a-Service observed 7.2.2 X-as-a-Service applied 7.3 Improvements with the concept of “Platform Architecture” 7.3.1 Platform Architecture Explained 7.3.2 Platform architecture applied 7.4 Improvements for the data producers 7.5 Summary Chapter 8: Self-Serve Data Platforms Applications in Comparison 8.1 Data Mesh on GCP 8.1.1 Self-Serve Data Platform architecture 8.1.2 Identifying the components of the Platform 8.1.3 Identifying the components of the Data Product 8.1.4 Workflows 8.1.5 Variations 8.1.6 Relation to Data Mesh ideas 8.1.7 GCP architecture summary 8.2 Data Mesh on AWS 8.2.1 Self-Serve Data Platform architecture 8.2.2 Identifying the components of the Platform 8.2.3 Identifying the components of the Data Products 8.2.4 Workflows 8.2.5 Relation to Data Mesh ideas 8.2.6 Variations 8.2.7 AWS architecture summary 8.3 Data Mesh on Databricks 8.3.1 Self-Serve Data Platform architecture 8.3.2 Identifying the components of the Platform 8.3.3 Identifying the components of the Data Product 8.3.4 Remarks 8.3.5 Variations 8.3.6 Databricks architecture summary 8.4 Data Mesh on Kafka 8.4.1 Self-Serve Data Platform architecture 8.4.2 Identifying the components 8.4.3 Remarks 8.4.4 Kafka Architecture summary 8.5 Summary Chapter 9: Solution architecture design 9.1 Capture and understand current architecture 9.1.1 What is software architecture? 9.1.2 How to document architecture: The C4 model 9.1.3 Looking at the C4 model 9.2 Architectural drivers of a Data Product design 9.2.1 Architectural drivers Functional requirements Quality attributes Constraints Principles 9.2.2 Capturing architectural drivers for a Data Product design Analyzing functional requirements Production Cost Statement Data Product requirements Choose the most important quality attributes and decide on the metric Look for constraints, principles, and quality attributes on the organizational level 9.3 Design future architecture of a Data Product and related systems 9.3.1 Design session 9.3.2 File-based data product: Spreadsheet Cost Statement Data Product: Solution number 1 Cost Statement Data Product: Solution number 2 9.3.3 From monolith and microservice to a data product Cast Data Product Turning the database inside-out Scripts Data Product 9.3.4 Exposing data for stream processing and batch processing Event stream as a Data Product port Data Lake and Data Products 9.4 Summary Revolutionize the way your organization approaches data with a data mesh! This new decentralized architecture outpaces monolithic lakes and warehouses and can work for a company of any size.In Data Mesh in Action you will learn how to: Implement a data mesh in your organization Turn data into a data product Move from your current data architecture to a data mesh Identify data domains, and decompose an organization into smaller, manageable domains Set up the central governance and local governance levels over data Balance responsibilities between the two levels of governance Establish a platform that allows efficient connection of distributed data products and automated governance Data Mesh in Action reveals how this groundbreaking architecture looks for both small startups and large enterprises. You won't need any new technology—this book shows you how to start implementing a data mesh with flexible processes and organizational change. You'll explore both an extended case study and multiple real-world examples. As you go, you'll be expertly guided through discussions around Socio-Technical Architecture and Domain-Driven Design with the goal of building a sleek data-as-a-product system. Plus, dozens of workshop techniques for both in-person and remote meetings help you onboard colleagues and drive a successful transition. About the technology Business increasingly relies on efficiently storing and accessing large volumes of data. The data mesh is a new way to decentralize data management that radically improves security and discoverability. A well-designed data mesh simplifies self-service data consumption and reduces the bottlenecks created by monolithic data architectures. About the book Data Mesh in Action teaches you pragmatic ways to decentralize your data and organize it into an effective data mesh. You'll start by building a minimum viable data product, which you'll expand into a self-service data platform, chapter-by-chapter. You'll love the book's unique “sliders” that adjust the mesh to meet your specific needs. You'll also learn processes and leadership techniques that will change the way you and your colleagues think about data. What's inside Decompose an organization into manageable domains Turn data into a data product Set up central and local governance levels Build a fit-for-purpose data platform Improve management, initiation, and support techniques About the reader For data professionals. Requires no specific programming stack or data platform. About the author Jacek Majchrzak is a hands-on lead data architect. Dr. Sven Balnojan manages data products and teams. Dr. Marian Siwiak is a data scientist and a management consultant for IT, scientific, and technical projects. Table of Contents PART 1 FOUNDATIONS 1 The what and why of the data mesh 2 Is a data mesh right for you? 3 Kickstart your data mesh MVP in a month PART 2 THE FOUR PRINCIPLES IN PRACTICE 4 Domain ownership 5 Data as a product 6 Federated computational governance 7 The self-serve data platform PART 3 INFRASTRUCTURE AND TECHNICAL ARCHITECTURE 8 Comparing self-serve data platforms 9 Solution architecture design