Data Analytics : Concepts, Techniques, and Applications

Large data sets arriving at every increasing speeds require a new set of efficient data analysis techniques. Data analytics are becoming an essential component for every organization and technologies such as health care, financial trading, Internet of Things, Smart Cities or Cyber Physical Systems. However, these diverse application domains give rise to new research challenges. In this context, the book provides a broad picture on the concepts, techniques, applications, and open research directions in this area. In addition, it serves as a single source of reference for acquiring the knowledge on emerging Big Data Analytics technologies-- Read more... Abstract: Large data sets arriving at every increasing speeds require a new set of efficient data analysis techniques. Data analytics are becoming an essential component for every organization and technologies such as health care, financial trading, Internet of Things, Smart Cities or Cyber Physical Systems. However, these diverse application domains give rise to new research challenges. In this context, the book provides a broad picture on the concepts, techniques, applications, and open research directions in this area. In addition, it serves as a single source of reference for acquiring the knowledge on emerging Big Data Analytics technologies Cover Half Title Title Page Copyright Page Dedication Contents Acknowledgments Preface List of Contributors SECTION I: DATA ANALYTICS CONCEPTS 1 An Introduction to Machine Learning 1.1 A Definition of Machine Learning 1.1.1 Supervised or Unsupervised? 1.2 Artificial Intelligence 1.2.1 The First AI Winter 1.3 ML and Statistics 1.3.1 Rediscovery of ML 1.4 Critical Events: A Timeline 1.5 Types of ML 1.5.1 Supervised Learning 1.5.2 Unsupervised Learning 1.5.3 Semisupervised Learning 1.5.4 Reinforcement Learning 1.6 Summary 1.7 Glossary References 2 Regression for Data Analytics 2.1 Introduction 2.1.1 Chapter Roadmap 2.1.2 What Is Regression? 2.2 Linear Regression 2.2.1 Dataset Description 2.2.2 Problem Definition 2.2.2.1 To Wrap Up the Whole Thing 2.2.3 Probabilistic Interpretation 2.2.4 Optimization Method 2.2.5 Block Diagram 2.2.6 Overview of the Model 2.3 Logistic Regression 2.3.1 Problem Definition 2.3.2 Logistic Function 2.3.3 Probabilistic Interpretation 2.3.4 Optimization Method 2.3.5 Overview of the Model 2.4 Problems of Regression 2.4.1 Underfitting and Overfitting 2.4.2 Outlier 2.4.3 Hyper-Parameter 2.5 Conclusion References 3 Big Data-Appropriate Clustering via Stochastic Approximation and Gaussian Mixture Models 3.1 Introduction 3.1.1 Chapter Roadmap 3.2 Stochastic Approximation Algorithm 3.2.1 Convergence Result 3.3 Gaussian Mixture Model 3.4 An SAA for Maximum Likelihood Estimation of GMMs 3.5 Simulation Results 3.5.1 Practical Considerations 3.5.2 Simulating Gaussian Mixture Models 3.5.3 Comparisons 3.5.4 Results 3.6 MNIST Application 3.6.1 Data Preprocessing 3.6.2 Results 3.7 Conclusions References 4 Information Retrieval Methods for Big Data Analytics on Text 4.1 Introduction to Information Retrieval 4.2 Vector Space Models 4.2.1 Document-Term Matrix 4.2.2 Distance Metrics 4.2.2.1 Euclidean Distance 4.2.2.2 Mahalanobis Distance 4.2.2.3 Jaccard Index 4.2.2.4 Cosine Similarity 4.2.2.5 Word Mover’s Distance 4.2.3 Term Frequency–Inverse Document Frequency 4.3 Information Retrieval Approaches 4.3.1 Latent Semantic Analysis 4.3.2 word2vec 4.3.2.1 CBOW Model 4.3.2.2 Skip-Gram Model 4.3.3 fastText 4.4 Walk-Through of IR: An Illustrative Example 4.5 Applications 4.5.1 Sentiment Extraction 4.5.2 Text Categorization/Spam Detection 4.5.3 Translation to Other Languages 4.5.4 Automated Q&A and Chatbots 4.5.5 Text Summarization 4.5.6 Resume Short Listing 4.5.7 Replacing Medical Codes from Patient’s Prescription 4.5.8 Writing Automatically/Text Generation/Poetry 4.6 Conclusions References 5 Big Graph Analytics 5.1 Introduction 5.1.1 Motivation and Challenges of Big Graph Analytics 5.1.2 Frameworks for Big Graph Analytics 5.1.3 Organization and Goal of This Chapter 5.2 Distributed Frameworks for Analyzing Big Static Graphs 5.2.1 Vertex-Centric Frameworks 5.2.1.1 Classification of Vertex-Centric Frameworks 5.2.2 Block-Centric Frameworks 5.2.2.1 BLOGEL 5.2.2.2 GIRAPH++ 5.2.3 Subgraph-Centric Frameworks 5.2.3.1 NScale 5.2.4 Matrix-Based Frameworks 5.2.4.1 PEGUSUS 5.2.5 DBMS-Based Frameworks 5.2.5.1 Pregelix 5.2.5.2 DG-SPARQL 5.3 Single-Machine Frameworks for Analyzing Big Static Graphs 5.4 Distributed Frameworks for Analyzing Big Dynamic Graphs 5.4.1 Frameworks for Analyzing Temporal Graphs 5.4.1.1 DeltaGraph 5.4.1.2 Chronos 5.4.2 Frameworks for Analyzing Streaming Graphs 5.5 Single-Machine Frameworks for Analyzing Big Dynamic Graphs 5.5.1 LLAMA and SLOTH 5.5.2 STINGER 5.6 Conclusions Notes References SECTION II: DATA ANALYTICS TECHNIQUES 6 Transition from Relational Database to Big Data and Analytics 6.1 Introduction 6.1.1 Background, Motivation, and Aim 6.1.2 Chapter Organization 6.2 Transition from Relational Database to Big Data 6.2.1 Relational Database 6.2.2 Introduction to Big Data 6.2.3 Relational Data vs. Big Data 6.3 Evolution of Big Data 6.3.1 Facts and Predictions about the Data Generated 6.3.2 Applications of Big Data 6.3.3 Fundamental Principle and Properties of Big Data 6.3.3.1 Issues with Traditional Architecture for Big Data Processing 6.3.3.2 Fundamental Principle for Scalable Database System 6.3.3.3 Properties of Big Data System 6.3.4 Generalized Framework for Big Data Processing 6.3.4.1 Storage and Precomputation Layer 6.3.4.2 Knowledge Discovery Layer (Serving Layer) 6.3.4.3 Real-Time Data Processing Layer (Speed Layer) 6.4 Big Data Analytics 6.4.1 Big Data Characteristics and Related Challenges 6.4.1.1 Volume 6.4.1.2 Velocity 6.4.1.3 Variety 6.4.2 Why Big Data Analytics? 6.4.2.1 Text Analytics 6.4.2.2 Audio Analytics 6.4.2.3 Video Analytics 6.4.2.4 Social Media Analytics 6.4.2.5 Predictive Analytics 6.4.3 Challenges in Big Data Analytics 6.4.3.1 Collect and Store Data 6.4.3.2 Data Management 6.4.3.3 Data Analysis 6.4.3.4 Security for Big Data 6.4.3.5 Visualization of Data 6.5 Tools and Technologies for Big Data Processing 6.5.1 Tools 6.5.1.1 Thrift 6.5.1.2 ZooKeeper 6.5.1.3 Hadoop DFS 6.5.2 Resource Management 6.5.3 NoSQL Database: Unstructured Data Management 6.5.3.1 Apache HBase 6.5.3.2 Apache Cassandra 6.5.4 Data Processing 6.5.4.1 Batch Processing 6.5.4.2 Distributed Stream Processing 6.5.4.3 Graph Processing 6.5.4.4 High-Level Languages for Data Processing 6.5.5 Data Analytics at the Speed Layer 6.6 Future Work and Conclusion 6.6.1 Future Work on Real-Time Data Analytics 6.6.2 Conclusion References 7 Big Graph Analytics: Techniques, Tools, Challenges, and Applications 7.1 Introduction 7.2 Graph + Big Data = Big Graph 7.2.1 The Scale of Big Graph: How Big Is Big Graph? 7.2.2 V’s of Big Graph 7.2.3 Graph Databases 7.3 Big Graph Analytics 7.3.1 Definition 7.3.2 Relationships: The Basics of Graph Analytics 7.4 Big Graph Analytics Approaches 7.4.1 In-Memory Big Graph Analytics 7.4.2 SSD-Based Big Graph Analytics 7.4.3 Disk-Based Big Graph Analytics 7.4.4 Other Big Graph Analytics Frameworks 7.5 Graph Analytic Techniques 7.5.1 Centrality Analysis 7.5.1.1 Degree Centrality 7.5.1.2 Eigenvector Centrality 7.5.1.3 Katz Centrality 7.5.1.4 PageRank Centrality 7.5.1.5 Closeness Centrality 7.5.1.6 Betweenness Centrality 7.5.2 Path Analysis 7.5.3 Community Analysis 7.5.4 Connectivity Analysis 7.6 Algorithms for Big Graph Analytics 7.6.1 PageRank 7.6.2 Connected Component 7.6.3 Distributed Minimum Spanning Tree 7.6.4 Graph Search 7.6.5 Clustering 7.7 Issues and Challenges of Big Graph Analytics 7.7.1 High-Degree Vertex 7.7.2 Sparseness 7.7.3 Data-Driven Computations 7.7.4 Unstructured Problems 7.7.5 In-Memory Challenge 7.7.6 Communication Overhead 7.7.7 Load Balancing 7.8 Applications of Big Graph Analytics 7.8.1 Social Network Analysis 7.8.2 Behavior Analytics 7.8.3 Biological Networks 7.8.4 Recommendation Systems 7.8.5 Smart Cities 7.8.6 Geospatial Data and Logistics 7.8.7 Insurance Fraud Detection 7.9 Conclusions References 8 Application of Game Theory for Big Data Analytics 8.1 Introduction 8.1.1 Chapter Roadmap 8.2 Basics of Classical and Evolutionary Game Theory 8.2.1 Classical Game Theory 8.2.2 Evolutionary Game Theory 8.2.3 Nash Equilibrium 8.2.4 Pareto Efficiency 8.2.5 Repeated Game 8.2.6 Bayesian Game 8.2.7 Chicken Game 8.2.8 Tit-for-Tat Game 8.2.9 Stackelberg Game 8.2.10 Potential Game 8.3 Game-Theoretic Application in Big Data Analytics 8.4 Limitations and Future Work 8.5 Conclusion References 9 Project Management for Effective Data Analytics 9.1 Introduction 9.1.1 Chapter Roadmap 9.2 Big Data Projects 9.3 Project Management Body of Knowledge 9.4 Projects in Controlled Environment 2 9.5 Agile 9.6 ISO 21500:2012 9.7 Key Insights 9.8 Conclusion References 10 Blockchain in the Era of Industry 4.0 10.1 Introduction 10.1.1 Chapter Roadmap 10.2 Emergence of Industrial Revolutions 10.2.1 Fourth Industrial Revolution (Industry 4.0) 10.2.2 Definition of Industry 4.0 10.2.3 Core Components of Industry 4.0 10.3 Blockchain and Cryptocurrency 10.3.1 Definition of Blockchain 10.3.2 Components of Blockchain 10.3.3 Working Procedure and Algorithm 10.3.4 Cryptocurrency 10.4 Blockchain’s Impact on Industry 4.0 10.4.1 How Blockchain Supports Industry 4.0 10.4.2 Application Domains of Blockchain in Industry 4.0 10.4.3 Adaptation Issues and Open Research Challenges 10.4.4 Challenges Associated with Law, Policy, and Standardization 10.4.5 Recommendations for Adaptation 10.5 Potential Use Case and Comparative Analysis 10.5.1 Use Case: DSC 10.5.2 Comparative Analysis 10.6 Conclusion References 11 Dark Data for Analytics 11.1 Introduction 11.1.1 Chapter Roadmap 11.2 Origin of Dark Data 11.3 Risks of Dark Data 11.4 Dark Data Analytics: An Untapped Opportunity 11.4.1 Implication of Dark Data in the Health Sector 11.4.2 Dark Data for Gaining Market Advantage 11.4.3 Dark Data for Social Media Insights 11.4.4 Retailers Providing Personalization with the Help of Dark Data 11.5 Different Ways to Eliminate Dark Data 11.5.1 Tools and Technique for Collecting and Analyzing Dark Data 11.5.2 A Brief Introduction to DeepDive 11.5.3 Six Steps to Identify and Manage Dark Data 11.6 Dark Data Solution Provided by Companies 11.6.1 AI Foundry’s Agile Solutions for Transformation of Dark Data 11.6.2 Dark Data Fracking by Datumize 11.6.3 Nuix Information Governance Solution 11.6.4 Deloitte: Insight’s Way to Start Extracting Value from Dark Data 11.7 International Data Corporation’s Research on Organization’s Ability to Derive Value from Dark Data 11.8 Recommendations on Managing Dark Data 11.9 Conclusion References SECTION III: DATA ANALYTICS APPLICATIONS 12 Big Data: Prospects and Applications in the Technical and Vocational Education and Training Sector 12.1 Introduction 12.1.1 What Is Big Data? 12.1.2 Chapter Roadmap 12.2 Big Data Technologies 12.2.1 Big Data Architecture Framework 12.2.2 Big Data Learning Experience Cycle 12.2.3 Benefits of Big Data 12.2.3.1 Enabling Personalized Learning 12.2.3.2 Proper Decision-Making 12.2.3.3 Measure Return on Investment 12.2.3.4 Performance Prediction 12.2.3.5 Determination of Student Behavior 12.3 Tools, Algorithms, and Analytic Platforms for Educational Purposes 12.4 Recommendation and Conclusion References 13 Sports Analytics: Visualizing Basketball Records in Graphical Form 13.1 Introduction 13.1.1 Chapter Roadmap 13.2 Background and Related Work 13.3 Design Details 13.3.1 Converting Text Format Data 13.3.2 Drawing Charts and Graphs 13.3.2.1 Line Chart 13.3.2.2 Track Lines 13.3.2.3 Dynamic Elements 13.3.3 Technologies 13.3.4 System Usage 13.3.4.1 General Usage 13.3.4.2 Specific Usage 13.4 User Study 13.4.1 Design and Participants 13.4.2 Measures 13.4.2.1 Search Efficiency and Clarity 13.4.2.2 Usability and Learnability 13.4.2.3 Visual Appeal 13.4.2.4 User Experience and Improvement 13.4.3 Materials and Apparatus 13.4.4 Procedures 13.5 Results 13.5.1 Search and Clarity 13.5.2 Usability and Learnability 13.5.3 Visual Appeal 13.5.4 Overall Experience 13.5.5 Subjective Report 13.6 Discussion 13.6.1 Limitations and Future Work 13.7 Conclusions Acknowledgments Declaration of Conflicting Interests Funding Notes References 14 Analysis of Traffic Offenses in Transportation: Application of Big Data Analysis 14.1 Introduction 14.1.1 Chapter Roadmap 14.2 Material and Methods 14.2.1 Data Inclusion Criteria 14.2.2 Data Preprocessing 14.2.3 Data Analysis 14.2.3.1 Linear Regression 14.2.3.2 Nonlinear (Polynomial) Regression 14.2.4 Data Analysis Algorithms 14.2.5 Statistical Analysis 14.3 Results 14.3.1 Top 15 Traffic Offenses 14.3.2 Directly Time-Related Offenses 14.3.3 Offenses against Year of Occurrence 14.3.4 Offenses against Month of Occurrence 14.3.4.1 Regression Model 14.3.4.2 Top 15 Traffic Offense Frequencies against Month 14.3.4.3 Obtained Results 14.3.5 Offenses against Weekday of Occurrence 14.3.5.1 Regression Model 14.3.5.2 Top 15 Traffic Offense Frequency against the Day of the Week 14.3.5.3 Obtained Results 14.3.6 Offenses against Time of Occurrence 14.3.6.1 Regression Model 14.3.6.2 Top 15 Traffic Offense Frequency against Time Period 14.3.6.3 Obtained Results 14.3.7 Summary of the Proposed Regression Models 14.4 Discussion 14.5 Conclusion References 15 Intrusion Detection for Big Data 15.1 Big Data and Intrusion Detection System 15.1.1 Chapter Roadmap 15.2 What is Big Data? 15.3 Security Issues with Big Data 15.4 Intrusion Detection System 15.5 Classification of Intrusion Detection Systems 15.5.1 Location-Based Classification 15.5.2 Evaluation Criteria-Based Classification 15.6 Collaborative Intrusion Detection and Big Data 15.6.1 Why Is It Necessary? 15.7 Architecture of CIDS 15.7.1 Centralized Architecture 15.7.2 Hierarchical Architecture 15.7.3 Distributed Architecture 15.8 Building Blocks of CIDS 15.8.1 Local Monitoring 15.8.2 Membership Management 15.8.3 Correlation and Aggregation 15.8.4 Data Dissemination 15.8.5 Global Monitoring 15.9 Attacks on Collaborative Intrusion System 15.9.1 External Attacks 15.9.1.1 Disclosure Attack 15.9.2 Evasion Attack 15.9.2.1 Internal Attack 15.10 Cloud Framework and Collaborative Intrusion Detection System 15.11 Coordinated Attacks 15.11.1 Large-Scale Stealthy Scans 15.11.2 Worm Outbreaks 15.11.3 Distributed Denial-of-Service Attacks 15.12 State-of-the-Art Existing Literatures 15.13 Future Direction and Conclusion References 16 Health Care Security Analytics 16.1 Introduction 16.1.1 Chapter Roadmap 16.2 Health Care in the Era of Industry 4.0 16.3 Taxonomy of Cyberattacks in The Health Care Domain 16.3.1 Attacks on Medical Devices 16.3.1.1 Magnetic Resonance Imaging (MRI) 16.3.1.2 Robotic Surgical Machine 16.3.1.3 Active Patient Monitoring Devices 16.3.2 Cyber-Physical Attacks 16.3.2.1 Attacks on Building Controls System 16.3.3 Insider Threat 16.4 Hacker’s Entry 16.4.1 Reconnaissance 16.4.1.1 Footprinting 16.4.1.2 Network Mapping 16.4.1.3 Scanning 16.4.2 Hacker’s Access Hospital Network 16.4.2.1 Phishing Attack 16.4.2.2 Ransomware 16.4.2.3 USB Stick 16.4.2.4 Password Cracker 16.4.2.5 Black Hole Attack 16.4.2.6 Rogue Access Points 16.5 Countermeasures 16.6 Conclusions References Index Large data sets arriving at every increasing speeds require a new set of efficient data analysis techniques. Data analytics are becoming an essential component for every organization and technologies such as health care, financial trading, Internet of Things, Smart Cities or Cyber Physical Systems. However, these diverse application domains give rise to new research challenges. In this context, the book provides a broad picture on the concepts, techniques, applications, and open research directions in this area. In addition, it serves as a single source of reference for acquiring the knowledge on emerging Big Data Analytics technologies-- Provided by publisher