On-Chip Training NPU - Algorithm, Architecture and SoC Design

Unlike most available sources that focus on deep neural network (DNN) inference, this book provides readers with a single-source reference on the needs, requirements, and challenges involved with on-device, DNN training semiconductor and SoC design. The authors include coverage of the trends and history surrounding the development of on-device DNN training, as well as on-device training semiconductors and SoC design examples to facilitate understanding. Abstract Keywords Contents List of Figures List of Tables 1 Introduction 1.1 Necessity of On-Device Deep Neural Network Training 1.2 Major Applications of On-Device DNN Training 1.2.1 Evolution: Long-Term DNN Training 1.2.2 Advancement: Medium-Term DNN Training 1.2.3 Adaptation: Short-Term DNN Training 1.2.4 Comparison of Three Application Scenarios 1.3 Contribution of This Book 2 A Theoretical Study on Artificial Intelligence Training 2.1 Types of Machine Learning 2.1.1 Supervised Learning Semi-supervised Learning Weakly Supervised Learning Self-supervised Learning Imitation Learning 2.1.2 Unsupervised Learning 2.1.3 Reinforcement Learning 2.1.4 Knowledge-Transfer-Based Learning Transfer Learning Domain Adaptation Knowledge Distillation 2.1.5 Learning According to Run-Time Data Collection: Active Learning vs. Online Learning Active Learning Online Learning 2.1.6 Multi-task Learning Fine-Tuning Joint Training Learning Without Forgetting 2.2 Universal Approximation Theorems 2.3 Perceptron 2.3.1 Basic Structure of Perceptron 2.3.2 Non-linear Activation Function 2.3.3 Multi-layer Perceptron (MLP) 2.4 DNN Training with Back-Propagation 2.4.1 Forward and Backward Paths of the Back-Propagation 2.4.2 Gradient Descent Numerical Gradient Descent Analytic Gradient Descent 2.4.3 Back-Propagation in Convolutional Neural Network 2.5 Brief Summary of Required Computations During BP 2.5.1 Back-Propagation of Fully Connected Layer and Convolution Layer Back-Propagation of Fully Connected Layer Back-Propagation of Convolutional Layer 2.6 Challenges of BP at the Micro-AI Systems 2.6.1 Reading of Transposed Weight 2.6.2 Maximizing Throughput Even with the Limited Hardware Resources 2.6.3 High-Bit-Precision Requirement 2.6.4 Memory-Intensive Operations During Weight-Gradient-Update 2.6.5 Backward Locking Problem 2.7 Biological Plausibility of Back-Propagation 2.8 DNN Training Without Back-Propagation 2.8.1 Genetic Algorithm Evolution Strategy 2.8.2 Local Learning Eligibility Propagation (E-prop) Hebbian Learning Predictive Coding Local Error Training Error Correction Learning Boltzmann Learning 2.8.3 Semi-back-propagation Feedback Alignment Direct Feedback Alignment Decoupled Delayed Gradient Features Replay Decoupled Neural Interface Fully Decoupled Gradient 2.9 Relationship Between Cloud Server and Mobile Devices During the Learning 2.9.1 3 Types of Learning Based on Its Training Platforms Server-Only Learning On-Device (Device-Only) Learning Server–Device Co-learning 2.9.2 Collaborative Learning Distributed Learning: Task Division and Aggregation Low-Cost Gradient Transmission Methodologies Federated Learning Gossip Training Temporal Knowledge Distillation 3 New Algorithm 1: Binary Direct Feedback Alignment for Fully Connected Layer 3.1 Introduction 3.2 Preliminaries 3.2.1 Back-propagation 3.2.2 Feedback Alignment 3.2.3 Direct Feedback Alignment (DFA) 3.3 Our Approach 3.3.1 CDFA: CNN Training by Combining both BP and DFA 3.3.2 Binary Direct Feedback Alignment (BDFA) 3.4 Experiments 3.4.1 CNN Training from the Scratch 3.4.2 Example of Online Learning with Small Dataset 3.5 Conclusion 4 New Algorithm 2: Extension of Direct Feedback Alignment to Convolutional Recurrent Neural Network 4.1 Introduction 4.2 Background 4.2.1 Back-propagation-based CNN and RNN Training 4.2.2 Direct Feedback Alignment based CNN and RNNTraining 4.3 Our Approach 4.3.1 Network Modularization 4.3.2 Computational Optimization of CNN Dilated-Convolution-Based Error Propagation Division of Channels During Error Propagation 4.3.3 Computational Optimization of RNN Revision of Backward Weight into Upper Triangular Matrix Highly Sparse and Binary Backward Weight 4.3.4 Hybrid Direct Feedback Alignment 4.4 Experiments 4.4.1 CNN Training Result 4.4.2 RNN Training Result 4.4.3 Summary of the HDFA based DNN Training 4.5 Conclusion 4.6 Appendices 4.6.1 Direct Feedback Alignment Based Convolutional Neural Network Training Synergy of the DFA with BP Scalability Issue of the DFA 4.6.2 Direct Feedback Alignment Based Recurrent Neural Network Training Synergy of the DFA with BP Backward Weight Initialization Method Effect of Backward Weight Sparsity 5 DF-LNPU: A Pipelined Direct Feedback Alignment Based Deep Neural Network Learning Processor for Fast Online Learning 5.1 Introduction 5.2 Related Algorithms 5.2.1 Back-Propagation and Backward Locking Problem 5.2.2 Direct Feedback Alignment 5.3 Pipelined Direct Feedback Alignment 5.4 Overall Architecture 5.5 Proposed Processor: DF-LNPU 5.5.1 Heterogeneous Learning Core Architecture Zero-Skip Based Convolution Layer Optimized Core PDFA Based Fully Connected Layer Learning Core 5.5.2 Inter-core and Intra-core Pipeline Design Inter-core Pipeline Design Intra-core Pipeline Design 5.5.3 Direct Error Propagation Core 5.6 Measurement Results 5.7 Conclusion 6 HNPU-V1: An Adaptive DNN Training Processor Utilizing Stochastic Dynamic Fixed-Point and Active Bit-Precision Searching 6.1 Introduction 6.2 Conventional Low-Bit-Precision Training Methods 6.2.1 Floating-Point Based Training 6.2.2 Fixed-Point Representation Based Training 6.3 Proposed Low-Bit-Precision Training Method 6.3.1 Stochastic Thresholding and Stochastic Dynamic Fixed-Point Representation 6.3.2 Layer-wise Adaptive Precision Scaling (LAPS) 6.4 Overall Architecture 6.4.1 Main Training Cores 6.4.2 Post-processing for Low-Precision Training 6.4.3 Task Allocation Methodologies for Training 6.4.4 Peripheral Components and Optimizer Core 6.5 Key Features of the Proposed Processor 6.5.1 Bit-Slice Serial Architecture with SDFXP and LAPS Unit Bit-Slice Training Unit with Bit-Slice Serial Computing SDFXP Unit with Pseudo-Random Number Generator LAPS Unit 6.5.2 In–Out Slice Skipping Core Architecture Slice-Level Input Skipping Precision-Aware Output Skipping In–Out Slice Skipping Core Architecture 6.5.3 Adaptive Bandwidth Reconfigurable AccumulationNetwork 6.6 Measurement Results 6.6.1 Chip Implementation Results 6.6.2 CNN Training Benchmark Results 6.6.3 Online Learning Demonstration Result 6.7 Conclusion 7 HNPU-V2: An Energy-Efficient DNN Training Processor for Robust Object Detection with Real-World EnvironmentalAdaptation 7.1 Introduction 7.2 Overall Architecture 7.3 Key Features of the Proposed Processor 7.3.1 Intrinsic True Random Number Generator (iTRNG) iTRNG for DNN iTRNG for Stochastic Rounding 7.3.2 Versatile Sparsity Exploitation Core Workload Balancing with Sparsity Spreading Channel Removal with Pruning-Aware Channel Reordering Unit 7.3.3 Multi-learning Task Allocation with LT-Flag-BasedControl 7.4 Measurement Results 7.4.1 Chip Implementation Results 7.4.2 Object Detection Demonstration Results 7.5 Conclusion 8 An Overview of Energy-Efficient DNN Training Processors 8.1 Solution of Transpose-Read During Error Propagation 8.1.1 Software-Level Solution 8.1.2 Architecture-Level Solution 8.1.3 Circuit-Level Solution 8.2 Sparsity Exploitation During DNN Training 8.2.1 Conventional Input/Weight Zero Skipping 8.2.2 ReLU-Aware Output Zero Skipping During the EP 8.2.3 Pruning-Aware Output Zero Skipping during the WG 8.2.4 In- and Out-Slice Skipping 8.3 Bit-Precision Optimization 8.3.1 New Number Representation New Floating-Point Representation New Fixed-Point Representation 8.3.2 Low-Bit-Precision Training Algorithm 8.3.3 Hardware Architecture for Bit-Precision Optimization Fused Multiply–Add Unit Active Training Supporting Unit 8.4 Weight-Gradient-Update Stage Acceleration 8.4.1 Homogeneous Core Approach 8.4.2 Heterogeneous Core Approach 8.4.3 Gradient Bit-Precision Reduction 8.4.4 Sparse Gradient Selection 8.4.5 Selective DNN Training 8.5 Backward Unlocking 8.5.1 Backward Unlocking with Delayed Back-Propagation 8.5.2 Direct Feedback Alignment Based Approaches 8.6 DNN Training Accelerator Examples 8.6.1 Applications and Examples of Training Processor Applications of DNN Training Examples of Training ProcessorDesign 8.6.2 Algorithm–Hardware Co-optimization in TrainingProcessor New Algorithm and Its Dedicated Hardware Hardware Performance Improvement Through the Algorithm Combination Relationship of Sparsity and Quantization During DNN Training Design Comparison: HNPU-V1 and HNPU-V2 8.7 Comparison of Inference and Training Processor Design 8.7.1 PE/Circuit-Level Differences 8.7.2 Architecture-Level Differences 8.7.3 Algorithm-Level Differences 8.8 Conclusion 9 Conclusion References Curriculum Vitae Curriculum Vitae Index