Data profiling refers to the activity of collecting data about data, i.e., metadata. Most IT professionals and researchers who work with data have engaged in data profiling, at least informally, to understand and explore an unfamiliar dataset or to determine whether a new dataset is appropriate for a particular task at hand. Data profiling results are also important in a variety of other situations, including query optimization, data integration, and data cleaning. Simple metadata are statistics, such as the number of rows and columns, schema and datatype information, the number of distinct values, statistical value distributions, and the number of null or empty values in each column. More complex types of metadata are statements about multiple columns and their correlation, such as candidate keys, functional dependencies, and other types of dependencies. This book provides a classification of the various types of profilable metadata, discusses popular data profiling tasks, and surveys state-of-the-art profiling algorithms. While most of the book focuses on tasks and algorithms for relational data profiling, we also briefly discuss systems and techniques for profiling non-relational data such as graphs and text. We conclude with a discussion of data profiling challenges and directions for future work in this area. Preface Acknowledgments Discovering Metadata Motivation and Overview Data Profiling and Data Mining Use Cases Organization of This Book Data Profiling Tasks Single-Column Analysis Dependency Discovery Relaxed Dependencies Single-Column Analysis Cardinalities Value Distributions Data Types, Patterns, and Domains Data Completeness Approximate Statistics Summary and Discussion Dependency Discovery Dependency Definitions Functional Dependencies Unique Column Combinations Inclusion Dependencies Search Space and Data Structures Lattices and Search Space Sizes Position List Indexes and Search Space Validation Search Complexity Null Semantics Discovering Unique Column Combinations Gordian HCA Ducc HyUCC Swan Discovering Functional Dependencies Tane Fun FD_Mine Dfd Dep-Miner FastFDs Fdep HyFD Discovering Inclusion Dependencies SQL-Based IND Validation B&B DeMarchi Binder Spider S-IndD Sindy Mind Find2 ZigZag Mind2 Relaxed and Other Dependencies Relaxing the Extent of a Dependency Partial Dependencies Conditional Dependencies Relaxing Attribute Comparisons Metric and Matching Dependencies Order and Sequential Dependencies Approximating the Dependency Discovery Generalizing Functional Dependencies Denial Constraints Multivalued Dependencies Use Cases Data Exploration Schema Engineering Data Cleaning Query Optimization Data Integration Profiling Non-Relational Data XML RDF Time Series Graphs Text Data Profiling Tools Research Prototypes Commercial Tools Data Profiling Challenges Functional Challenges Profiling Dynamic Data Interactive Profiling Profiling for Integration Interpreting Profiling Results Non-Functional Challenges Efficiency and Scalability Profiling on New Architectures Benchmarking Profiling Methods Conclusions Bibliography Authors' Biographies Blank Page Provides a classification of the various types of profilable metadata, discusses data profiling tasks, and surveys state-of-the-art profiling algorithms. While most of the book focuses on tasks and algorithms for relational data profiling, it also briefly discusses systems and techniques for profiling non-relational data such as graphs and text.
