Principles and Practice of Big Data : Preparing, Sharing, and Analyzing Complex Information

Principles and Practice of Big Data: Preparing, Sharing, and Analyzing Complex Information, Second Edition updates and expands on the first edition, bringing a set of techniques and algorithms that are tailored to Big Data projects. The book stresses the point that most data analyses conducted on large, complex data sets can be achieved without the use of specialized suites of software (e.g., Hadoop), and without expensive hardware (e.g., supercomputers). The core of every algorithm described in the book can be implemented in a few lines of code using just about any popular programming language (Python snippets are provided). Through the use of new multiple examples, this edition demonstrates that if we understand our data, and if we know how to ask the right questions, we can learn a great deal from large and complex data collections. The book will assist students and professionals from all scientific backgrounds who are interested in stepping outside the traditional boundaries of their chosen academic disciplines. Presents new methodologies that are widely applicable to just about any project involving large and complex datasets Offers readers informative new case studies across a range scientific and engineering disciplines Provides insights into semantics, identification, de-identification, vulnerabilities and regulatory/legal issues Utilizes a combination of pseudocode and very short snippets of Python code to show readers how they may develop their own projects without downloading or learning new software Preface......Page 3 Section 1.1. Definition of Big Data......Page 11 Section 1.2. Big Data Versus Small Data......Page 13 Section 1.3. Whence Comest Big Data?......Page 15 Section 1.4. The Most Common Purpose of Big Data Is to Produce Small Data......Page 17 Section 1.5. Big Data Sits at the Center of the Research Universe......Page 18 Glossary......Page 19 References......Page 23 Section 2.1. Nearly All Data Is Unstructured and Unusable in Its Raw Form......Page 24 Section 2.2. Concordances......Page 25 Section 2.3. Term Extraction......Page 28 Section 2.4. Indexing......Page 31 Section 2.5. Autocoding......Page 33 Section 2.6. Case Study: Instantly Finding the Precise Location of Any Atom in the Universe (Some Assembly Required)......Page 38 Section 2.7. Case Study (Advanced): A Complete Autocoder (in 12 Lines of Python Code)......Page 40 Section 2.8. Case Study: Concordances as Transformations of Text......Page 43 Section 2.9. Case Study (Advanced): Burrows Wheeler Transform (BWT)......Page 45 Glossary......Page 48 References......Page 59 Section 3.1. What Are Identifiers?......Page 61 Section 3.2. Difference Between an Identifier and an Identifier System......Page 63 Section 3.3. Generating Unique Identifiers......Page 66 Section 3.4. Really Bad Identifier Methods......Page 68 Section 3.5. Registering Unique Object Identifiers......Page 71 Section 3.6. Deidentification and Reidentification......Page 74 Section 3.7. Case Study: Data Scrubbing......Page 77 Section 3.8. Case Study (Advanced): Identifiers in Image Headers......Page 79 Section 3.9. Case Study: One-Way Hashes......Page 82 Glossary......Page 84 References......Page 90 Section 4.2. eXtensible Markup Language......Page 93 Section 4.3. Semantics and Triples......Page 95 Section 4.4. Namespaces......Page 96 Section 4.5. Case Study: A Syntax for Triples......Page 98 Section 4.6. Case Study: Dublin Core......Page 101 Glossary......Page 102 References......Page 103 Section 5.1. It’s All About Object Relationships......Page 104 Section 5.2. Classifications, the Simplest of Ontologies......Page 108 Section 5.3. Ontologies, Classes With Multiple Parents......Page 111 Section 5.4. Choosing a Class Model......Page 113 Section 5.5. Class Blending......Page 117 Section 5.6. Common Pitfalls in Ontology Development......Page 118 Section 5.7. Case Study: An Upper Level Ontology......Page 121 Section 5.8. Case Study (Advanced): Paradoxes......Page 122 Section 5.9. Case Study (Advanced): RDF Schemas and Class Properties......Page 124 Section 5.10. Case Study (Advanced): Visualizing Class Relationships......Page 127 Glossary......Page 132 References......Page 141 Section 6.1. Knowledge of Self......Page 143 Section 6.2. Data Objects: The Essential Ingredient of Every Big Data Collection......Page 146 Section 6.3. How Big Data Uses Introspection......Page 148 Section 6.4. Case Study: Time Stamping Data......Page 151 Section 6.5. Case Study: A Visit to the TripleStore......Page 153 Section 6.6. Case Study (Advanced): Proof That Big Data Must Be Object-Oriented......Page 158 Glossary......Page 159 References......Page 160 Section 7.1. Standards......Page 161 Section 7.2. Specifications Versus Standards......Page 166 Section 7.3. Versioning......Page 168 Section 7.4. Compliance Issues......Page 170 Section 7.5. Case Study: Standardizing the Chocolate Teapot......Page 171 Glossary......Page 172 References......Page 173 Section 8.1. The Importance of Data That Cannot Change......Page 174 Section 8.2. Immutability and Identifiers......Page 175 Section 8.3. Coping With the Data That Data Creates......Page 178 Section 8.4. Reconciling Identifiers Across Institutions......Page 179 Section 8.6. Case Study: Blockchains and Distributed Ledgers......Page 181 Section 8.7. Case Study (Advanced): Zero-Knowledge Reconciliation......Page 184 Glossary......Page 186 References......Page 188 Section 9.1. Looking at the Data......Page 189 Section 9.2. The Minimal Necessary Properties of Big Data......Page 196 Section 9.3. Data That Comes With Conditions......Page 201 Section 9.4. Case Study: Utilities for Viewing and Searching Large Files......Page 202 Section 9.5. Case Study: Flattened Data......Page 204 Glossary......Page 205 References......Page 209 Section 10.1. Accuracy and Precision......Page 211 Section 10.2. Data Range......Page 213 Section 10.3. Counting......Page 215 Section 10.4. Normalizing and Transforming Your Data......Page 219 Section 10.5. Reducing Your Data......Page 223 Section 10.6. Understanding Your Control......Page 226 Section 10.7. Statistical Significance Without Practical Significance......Page 227 Section 10.8. Case Study: Gene Counting......Page 228 Section 10.9. Case Study: Early Biometrics, and the Significance of Narrow Data Ranges......Page 229 Glossary......Page 230 References......Page 232 Section 11.1. Speed and Scalability......Page 234 Section 11.2. Fast Operations, Suitable for Big Data, That Every Computer Supports......Page 240 Section 11.3. The Dot Product, a Simple and Fast Correlation Method......Page 246 Section 11.4. Clustering......Page 248 Section 11.5. Methods for Data Persistence (Without Using a Database)......Page 250 Section 11.6. Case Study: Climbing a Classification......Page 252 Section 11.7. Case Study (Advanced): A Database Example......Page 254 Section 11.8. Case Study (Advanced): NoSQL......Page 255 Glossary......Page 256 References......Page 259 Section 12.1. Denominators......Page 261 Section 12.2. Word Frequency Distributions......Page 262 Section 12.3. Outliers and Anomalies......Page 266 Section 12.4. Back-of-Envelope Analyses......Page 268 Section 12.5. Case Study: Predicting User Preferences......Page 270 Section 12.6. Case Study: Multimodality in Population Data......Page 272 Glossary......Page 273 References......Page 277 Section 13.1. The Remarkable Utility of (Pseudo)Random Numbers......Page 278 Section 13.2. Repeated Sampling......Page 284 Section 13.3. Monte Carlo Simulations......Page 289 Section 13.4. Case Study: Proving the Central Limit Theorem......Page 292 Section 13.5. Case Study: Frequency of Unlikely String of Occurrences......Page 294 Section 13.6. Case Study: The Infamous Birthday Problem......Page 295 Section 13.7. Case Study (Advanced): The Monty Hall Problem......Page 296 Section 13.8. Case Study (Advanced): A Bayesian Analysis......Page 298 Glossary......Page 300 References......Page 302 Section 14.1. Theory in Search of Data......Page 304 Section 14.2. Data in Search of Theory......Page 305 Section 14.3. Bigness Biases......Page 306 Section 14.4. Data Subsets in Big Data: Neither Additive Nor Transitive......Page 311 Section 14.5. Additional Big Data Pitfalls......Page 312 Section 14.6. Case Study (Advanced): Curse of Dimensionality......Page 315 Glossary......Page 317 References......Page 319 Section 15.1. Failure Is Common......Page 321 Section 15.2. Failed Standards......Page 323 Section 15.3. Blaming Complexity......Page 326 Section 15.4. An Approach to Big Data That May Work for You......Page 328 Section 15.5. After Failure......Page 337 Section 15.6. Case Study: Cancer Biomedical Informatics Grid, a Bridge Too Far......Page 339 Section 15.7. Case Study: The Gaussian Copula Function......Page 344 Glossary......Page 345 References......Page 347 Section 16.1. First Analysis (Nearly) Always Wrong......Page 350 Section 16.2. Why Reanalysis Is More Important Than Analysis......Page 353 Section 16.3. Case Study: Reanalysis of Old JADE Collider Data......Page 355 Section 16.5. Case Study: Finding New Planets From Old Data......Page 356 References......Page 358 Section 17.1. What Is Data Repurposing?......Page 362 Section 17.2. Dark Data, Abandoned Data, and Legacy Data......Page 364 Section 17.3. Case Study: From Postal Code to Demographic Keystone......Page 366 Section 17.4. Case Study: Scientific Inferencing From a Database of Genetic Sequences......Page 367 Section 17.6. Case Study: Inferring Climate Trends With Geologic Data......Page 368 Section 17.7. Case Study: Lunar Orbiter Image Recovery Project......Page 369 Glossary......Page 370 References......Page 371 Section 18.1. What Is Data Sharing, and Why Don’t We Do More of It?......Page 372 Section 18.2. Common Complaints......Page 373 Section 18.3. Data Security and Cryptographic Protocols......Page 380 Section 18.4. Case Study: Life on Mars......Page 386 Section 18.5. Case Study: Personal Identifiers......Page 387 Glossary......Page 389 References......Page 390 Section 19.1. Responsibility for the Accuracy and Legitimacy of Data......Page 393 Section 19.2. Rights to Create, Use, and Share the Resource......Page 396 Section 19.3. Copyright and Patent Infringements Incurred by Using Standards......Page 398 Section 19.4. Protections for Individuals......Page 400 Section 19.5. Consent......Page 402 Section 19.6. Unconsented Data......Page 407 Section 19.7. Privacy Policies......Page 409 Section 19.8. Case Study: Timely Access to Big Data......Page 410 Section 19.9. Case Study: The Havasupai Story......Page 411 Glossary......Page 413 References......Page 414 Section 20.1. How Big Data Is Perceived by the Public......Page 416 Section 20.2. Reducing Costs and Increasing Productivity With Big Data......Page 419 Section 20.3. Public Mistrust......Page 421 Section 20.4. Saving Us From Ourselves......Page 422 Section 20.5. Who Is Big Data?......Page 425 Section 20.6. Hubris and Hyperbole......Page 431 Section 20.7. Case Study: The Citizen Scientists......Page 434 Section 20.8. Case Study: 1984, by George Orwell......Page 437 Glossary......Page 438 References......Page 439 Index......Page 442