Hardware-Aware Probabilistic Machine Learning Models : Learning, Inference and Use Cases

This book proposes probabilistic machine learning models that represent the hardware properties of the device hosting them. These models can be used to evaluate the impact that a specific device configuration may have on resource consumption and performance of the machine learning task, with the overarching goal of balancing the two optimally. The book first motivates extreme-edge computing in the context of the Internet of Things (IoT) paradigm. Then, it briefly reviews the steps involved in the execution of a machine learning task and identifies the implications associated with implementing this type of workload in resource-constrained devices. The core of this book focuses on augmenting and exploiting the properties of Bayesian Networks and Probabilistic Circuits in order to endow them with hardware-awareness. The proposed models can encode the properties of various device sub-systems that are typically not considered by other resource-aware strategies, bringing about resource-saving opportunities that traditional approaches fail to uncover. The performance of the proposed models and strategies is empirically evaluated for several use cases. All of the considered examples show the potential of attaining significant resource-saving opportunities with minimal accuracy losses at application time. Overall, this book constitutes a novel approach to hardware-algorithm co-optimization that further bridges the fields of Machine Learning and Electrical Engineering. Introduces a new, systematic approach for the realization of hardware-awareness with probabilistic models; Enables readers to accommodate various systems and applications, as demonstrated with multiple use cases targeting distinct types of devices; Describes novel methods to deal with some of the challenges of extreme-edge computing, a paradigm that has recently garnered attention as a complementary approach to cloud computing; Represents one of the first efforts systematically to bring probabilistic inference to the world of edge computing, by means of novel algorithmic insights and strategies Preface Acknowledgements Contents 1 Introduction 1.1 The IoT Paradigm 1.1.1 Moving Intelligence Towards the Extreme Edge 1.2 The Machine Learning Pipeline 1.2.1 Tasks in Machine Learning 1.2.2 Performance Evaluation 1.2.3 Models 1.2.4 Features 1.3 Resource-Efficient Machine Learning 1.3.1 Inference 1.3.2 Feature Extraction 1.3.3 Learning 1.3.4 Remaining Challenges 1.4 Problem Statement 1.5 Sketch of the Proposed Solution 1.5.1 The Choice for Probabilistic Models 1.5.2 System-Wide Hardware-Awareness 1.6 Structure of This Book References 2 Background 2.1 Probability Theory 2.1.1 Basic Notions and Notation 2.1.2 Probability Distributions 2.1.3 Probabilistic Inference 2.1.4 Independence Notions and Bayes Rule 2.2 Bayesian Networks 2.2.1 Exact Inference in Bayesian Networks 2.2.2 Parameter Learning 2.2.3 Bayesian Network Classifiers 2.3 Probabilistic Circuits 2.3.1 Properties of Probabilistic Circuits 2.3.2 Structural Constraints 2.3.3 Classification Tasks with Probabilistic Circuits 2.4 Sensory Embedded Pipeline 2.4.1 Building Blocks References 3 Hardware-Aware Cost Models 3.1 Hardware-Aware Cost 3.2 Sensing Costs 3.2.1 Resource Versus Quality Trade-Offs of Mixed-Signal Sensor Front-Ends 3.3 Feature Extraction Costs 3.3.1 Digital Feature Precision Scaling 3.3.2 Analog Feature Precision Scaling 3.4 Inference Costs 3.5 Dynamic Tuning Costs for Run-Time Strategies 3.6 Types of Systems Considered in This Book 3.7 Conclusion References 4 Hardware-Aware Bayesian Networks for Sensor Front-End Quality Scaling 4.1 Noise-Scalable Bayesian Network Classifier 4.1.1 Model 4.1.2 Inference 4.2 Local Pareto-Optimal Feature Quality Tuning 4.3 Use Cases of the ns-BN: Introduction 4.4 First Use Case: Mixed-Signal Quality Scaling 4.4.1 Experiments for Mixed-Signal Quality Scaling Cost Function and Objective Function Definition Experimental Setup Results 4.5 Second Use Case: Digital Quality Scaling 4.5.1 Experiments for Digital Quality Scaling Cost and Objective Function Definition Experimental Setup Results 4.6 Third Use Case: Analog Quality Scaling 4.6.1 Experiments for Analog Quality Scaling Cost and Objective Function Definition Experimental Setup Results 4.7 Related Work 4.8 Discussion References 5 Hardware-Aware Probabilistic Circuits 5.1 Preliminaries 5.1.1 Probabilistic Sentential Decision Diagrams LearnPSDD 5.2 Hardware-Aware System-Wide Cost 5.3 Pareto-Optimal Trade-off Extraction 5.3.1 Search Strategy 5.3.2 Pareto-Optimal Configuration Selection 5.4 Experiments: Pareto-Optimal Trade-off 5.4.1 Embedded Human Activity Recognition Pareto-Optimal Configuration Robustness Against Failing Features 5.4.2 Generality of the Method: Evaluation on Benchmark Datasets 5.5 Learning PSDDs with a Discriminative Bias for Classification Tasks 5.5.1 Discriminative Bias for PSDD Learning 5.5.2 Generative Bias and Vtree Learning 5.6 Experiments: Biased PSDD Learning 5.6.1 Experimental Setup 5.6.2 Evaluation of D-LearnPSDD 5.6.3 Impact of the Vtree on Discriminative Performance 5.6.4 Robustness to Missing Features 5.7 Related Work 5.8 Discussion References 6 Run-Time Strategies 6.1 Noise-Scalable Bayesian Networks: Missing Features at Run-Time 6.1.1 Run-Time Pareto-Optimal Selection 6.1.2 Experiments: Robustness to Missing Features of ns-BN 6.1.3 Digital Overhead Costs 6.1.4 Remaining Challenges 6.2 Run-Time Implementation of Hardware-Aware PSDDs: Introduction 6.3 Model Complexity Switching Strategy 6.3.1 Model Selection 6.3.2 Model Switching Policy Setting the Thresholds 6.3.3 Time Aspects Prediction and Policy Frequency Prediction Stability 6.3.4 Experiments for Model Switching: Introduction 6.3.5 Model Selection for Experiments 6.3.6 Hyperparameter Selection 6.3.7 Performance of the Model Switching Strategy 6.3.8 Robustness to Missing Features 6.4 Related Work 6.5 Discussion References 7 Conclusions 7.1 Overview and Contributions 7.2 Suggestions for Future Work 7.3 Closing Remark Reference A Features Used for Experiments A.1 Synthetic Dataset for Digital Scaling ns-BN A.2 Six-Class HAR Classification with nsbn B Full List of Hyperparameters for Feature Pruning C Full List of Hyperparameters for Precision Reduction Index