Deep Neural Evolution: Deep Learning with Evolutionary Computation (Natural Computing Series)

This book delivers the state of the art in deep learning (DL) methods hybridized with evolutionary computation (EC). Over the last decade, DL has dramatically reformed many domains: computer vision, speech recognition, healthcare, and automatic game playing, to mention only a few. All DL models, using different architectures and algorithms, utilize multiple processing layers for extracting a hierarchy of abstractions of data. Their remarkable successes notwithstanding, these powerful models are facing many challenges, and this book presents the collaborative efforts by researchers in EC to solve some of the problems in DL. EC comprises optimization techniques that are useful when problems are complex or poorly understood, or insufficient information about the problem domain is available. This family of algorithms has proven effective in solving problems with challenging characteristics such as non-convexity, non-linearity, noise, and irregularity, which dampen the performance of most classic optimization schemes. Furthermore, EC has been extensively and successfully applied in artificial neural network (ANN) research —from parameter estimation to structure optimization. Consequently, EC researchers are enthusiastic about applying their arsenal for the design and optimization of deep neural networks (DNN). This book brings together the recent progress in DL research where the focus is particularly on three sub-domains that integrate EC with DL: (1) EC for hyper-parameter optimization in DNN; (2) EC for DNN architecture design; and (3) Deep neuroevolution. The book also presents interesting applications of DL with EC in real-world problems, e.g., malware classification and object detection. Additionally, it covers recent applications of EC in DL, e.g. generative adversarial networks (GAN) training and adversarial attacks. The book aims to prompt and facilitate the research in DL with EC both in theory and in practice. Preface Contents Abbreviations Part I Preliminaries 1 Evolutionary Computation and Meta-heuristics 1.1 Introduction 1.2 Evolutionary Algorithms: From Bullet Trains to Finance and Robots 1.3 Multi-Objective Optimization 1.4 Genetic Programming and Its Genome Representation 1.4.1 Tree-based Representation of Genetic Programming 1.4.2 Cartesian Genetic Programming (CGP) 1.5 Ant Colony Optimization (ACO) 1.6 Particle Swarm Optimization (PSO) 1.7 Artificial Bee Colony Optimization (ABC) 1.8 Firefly Algorithms 1.9 Cuckoo Search 1.10 Harmony Search (HS) 1.11 Conclusion References 2 A Shallow Introduction to Deep Neural Networks 2.1 Introduction 2.2 (Shallow) Neural Networks 2.2.1 Backpropagation Algorithm for Training NNs 2.3 Deep Neural Networks: What, Why and How? 2.4 Architectures of Deep Networks 2.4.1 Convolutional Neural Network 2.4.1.1 Convolutional Layers 2.4.1.2 Pooling Layers 2.4.1.3 Fully Connected Layers 2.4.1.4 Training Strategies 2.4.1.5 Popular CNN Models 2.4.2 Recurrent Neural Network 2.4.2.1 RNN Architecture 2.4.2.2 RNN Training 2.4.2.3 Memory Cells 2.4.3 Deep Autoencoder 2.4.4 Deep Belief Network (DBN) 2.4.5 Generative Adversarial Network (GAN) 2.4.5.1 GAN Architecture 2.4.5.2 GAN Training 2.4.5.3 Progresses in GAN Research 2.4.6 Recursive Neural Networks 2.5 Applications of Deep Learning 2.6 Conclusion References Part II Hyper-Parameter Optimization 3 On the Assessment of Nature-Inspired Meta-Heuristic Optimization Techniques to Fine-Tune Deep Belief Networks 3.1 Introduction 3.2 Theoretical Background 3.2.1 Restricted Boltzmann Machines 3.2.2 Contrastive Divergence 3.2.3 Persistent Contrastive Divergence 3.2.4 Deep Belief Networks 3.3 Meta-heuristic Optimization Algorithms 3.4 Methodology 3.4.1 Modeling DBN Hyper-parameter Fine-tuning 3.4.2 Datasets 3.4.3 Experimental Setup 3.5 Experimental Results 3.5.1 Training Evaluation 3.5.2 Time Analysis 3.5.3 Hyper-Parameters Analysis 3.6 Conclusions and Future Works References 4 Automated Development of DNN Based Spoken Language Systems Using Evolutionary Algorithms 4.1 Spoken Language Processing Systems 4.1.1 Principle of Speech Recognition 4.1.2 Hidden Markov Model Based Acoustic Modeling 4.1.3 End-to-End Speech Recognition System 4.1.4 Evaluation Measures 4.2 Evolutionary Algorithms 4.2.1 Genetic Algorithm 4.2.2 Evolution Strategy 4.2.3 Bayesian Optimization 4.3 Multi-Objective Optimization with Pareto Optimality 4.3.1 Pareto Optimality 4.3.2 CMA-ES with Pareto Optimality 4.3.3 Alternative Multi-Objective Methods 4.4 Experimental Setups 4.4.1 General Setups 4.4.2 Automatic Optimizations 4.5 Results 4.6 Conclusion References 5 Search Heuristics for the Optimization of DBN for Time Series Forecasting 5.1 Introduction 5.2 DBNs with RBMs and MLP 5.2.1 RBM 5.2.2 Training of RBM and DBN 5.2.3 DBNs used in Time Series Forecasting 5.3 Design DBN with PSO or Random Search 5.3.1 Design DBN with PSO 5.3.2 Design DBN with RS 5.4 Experimental Comparison of PSO and RS 5.4.1 One-ahead Prediction of Lorenz Chaos 5.4.1.1 DBN Decided by PSO for Lorenz Chaos 5.4.1.2 DBN Decided by RS for Lorenz Chaos 5.4.2 One-ahead Prediction of Hnon Map 5.4.3 Long-term Prediction of CATS Benchmark 5.5 Advanced Forecasting Systems Using DBNs 5.6 Conclusion References Part III Structure Optimization 6 Particle Swarm Optimization for Evolving Deep Convolutional Neural Networks for Image Classification: Single- and Multi-Objective Approaches 6.1 Introduction 6.1.1 Goals 6.2 Background 6.2.1 CNN Architecture 6.2.2 Internet Protocol Address 6.2.3 DenseNet 6.2.4 OMOPSO 6.3 The Methods 6.3.1 Single-Objective PSO Method 6.3.1.1 Algorithm Overview 6.3.1.2 Particle Encoding Strategy to Encode CNN Layers 6.3.1.3 Pseudo-Variable-Length Representation by Introducing Disabled Layers 6.3.1.4 An Example of the Encoding Strategy 6.3.1.5 Population Initialization 6.3.1.6 Fitness Evaluation 6.3.1.7 Update Particle with Velocity Clamping 6.3.1.8 Best Individual Selection and Decoding 6.3.2 Multi-Objective PSO Method 6.3.2.1 Algorithm Overview 6.3.2.2 Particle Encoding Strategy 6.3.2.3 Population Initialization 6.3.2.4 Objective Evaluation 6.4 Experiment Design 6.4.1 Benchmark Datasets 6.4.2 Peer Competitors 6.4.3 Parameter Settings 6.5 Results and Analysis 6.5.1 Results and Analysis of Single-Objective PSO Method 6.5.1.1 Overall Performance 6.5.1.2 Evolved CNN Architectures 6.5.1.3 Visualization 6.5.2 Results and Analysis of Multi-Objective PSO Method 6.5.2.1 Pareto Optimality Analysis 6.5.2.2 MOCNN vs DenseNet-121 6.5.2.3 Computational Cost 6.6 Conclusions References 7 Designing Convolutional Neural Network Architectures Using Cartesian Genetic Programming 7.1 Introduction 7.2 Progress of Neural Architecture Search 7.3 Designing CNN Architecture for Image Classification 7.3.1 Representation of CNN Architectures 7.3.2 Evolutionary Algorithm 7.3.3 Experiment on Image Classification Tasks 7.3.3.1 Experimental Setting 7.3.3.2 Experimental Result 7.4 Designing CNN Architectures for Image Restoration 7.4.1 Search Space of Network Architectures 7.4.2 Representation of CAE Architectures 7.4.3 Experiment on Image Restoration Tasks 7.4.3.1 Experimental Settings 7.4.3.2 Results of the Inpainting Tasks 7.4.3.3 Results of the Denoising Task 7.4.3.4 Analysis of Optimized Architectures 7.5 Summary References 8 Fast Evolution of CNN Architecture for Image Classification 8.1 Introduction 8.2 A Brief Overview of CNNs 8.3 Related Works 8.4 The Proposed Genetic Algorithm for Designing CNNs 8.4.1 Population Initialization 8.4.2 Fitness Evaluation 8.4.3 Creating New Generation 8.5 Experimental Setup 8.5.1 Datasets 8.5.2 Experimental Environment 8.5.3 Parameter Selection 8.6 Experimental Results 8.7 Discussion 8.8 Conclusion and Future Work References Part IV Deep Neuroevolution 9 Discovering Gated Recurrent Neural Network Architectures 9.1 Introduction 9.2 Background and Related Work 9.3 Methods 9.3.1 Genetic Programming for Recurrent Nodes 9.3.2 Speciation and Crossover 9.3.3 Search Space: Node 9.3.4 Search Space: Network 9.3.5 Meta-LSTM for Fitness Prediction 9.4 Experiments 9.4.1 Natural Language Modeling Task 9.4.2 Music Modeling Task 9.4.3 Network Training Details 9.4.4 Experiment 1: Evolution of Recurrent Nodes 9.4.5 Experiment 2: Heterogeneous Recurrent Networks 9.4.6 Experiment 3: Music Modeling 9.5 Discussion and Future Work 9.6 Conclusion References 10 Investigating Deep Recurrent Connections and Recurrent Memory Cells Using Neuro-Evolution 10.1 Introduction 10.2 Related Work 10.2.1 Elman, Jordan and Arbitrary Recurrent Connections 10.2.2 Recurrent Memory Cells 10.2.3 Temporal Skip Connections 10.2.4 Evolving Recurrent Neural Networks 10.3 Evolutionary eXploration of Augmenting Memory Models 10.3.1 Mutation and Recombination Operations 10.3.1.1 Edge Mutations 10.3.1.2 Node Mutations 10.3.1.3 Other Operations 10.3.2 Lamarckian Weight Initialization 10.4 Datasets 10.4.1 Aviation Flight Recorder Data 10.4.2 Coal-Fired Power Plant Data 10.5 Results 10.5.1 Experiments 10.5.2 EXAMM and Backpropagation Hyperparameters 10.5.3 Experimental Results 10.5.4 Memory Cell Performance 10.5.5 Effects of Simple Neurons 10.5.6 Effects of Deep Recurrent Connections 10.6 Discussion 10.7 Future Work 10.8 Conclusions References 11 Neuroevolution of Generative Adversarial Networks 11.1 Introduction 11.2 Generative Adversarial Networks 11.2.1 Definition 11.2.2 Common Problems in GAN Training 11.2.2.1 Mode Collapse 11.2.2.2 Vanishing Gradient 11.2.3 Evaluation Metrics 11.2.3.1 Inception Score 11.2.3.2 Fréchet Inception Distance 11.3 Exploring the Evolution of GANs 11.3.1 Neuroevolution 11.3.2 Variations of GANs 11.4 Current Proposals 11.4.1 E-GAN 11.4.2 Pareto GAN 11.4.3 Lipizzaner 11.4.4 Mustangs 11.4.5 COEGAN 11.5 Discussion 11.5.1 Characteristics of the GAN Model 11.5.2 Aspects of the Evolutionary Algorithm 11.5.3 Experiments and Results 11.6 Conclusions References Part V Applications and Others 12 Evolving Deep Neural Networks for X-ray Based Detection of Dangerous Objects 12.1 Introduction 12.2 Related Research 12.2.1 Neuro-Evolution 12.2.2 Genetic CNN 12.2.3 Aggressive Selection and Mutation 12.2.4 YOLO 12.2.5 Transfer Learning 12.3 Proposed Method 12.4 Experiments on Evolutionary Synthesis of Convolutional Neural Networks 12.4.1 MNIST Handwritten Number Classification Experiment 12.4.1.1 Initial Generation 12.4.1.2 Mutation Operations 12.4.1.3 Experimental Results 12.4.2 Experiment on Detecting Dangerous Objects in X-ray Images 12.4.2.1 Initial Generation 12.4.2.2 Mutation Operations 12.4.2.3 Restructuring of Network Structures Due to Mutation 12.4.2.4 Fitness (mAP) Calculation Method 12.4.2.5 Changing the Evaluation Method for Individuals in the Elite Group of Each Generation 12.4.2.6 Experimental Results 12.5 Discussion 12.5.1 Handwritten Number Classification Experiment Using MNIST 12.5.2 Experiment on Detecting Dangerous Objects in X-ray Images 12.6 Conclusion References 13 Evolving the Architecture and Hyperparameters of DNNs for Malware Detection 13.1 Introduction 13.2 Deep Learning Based Android Malware Detection and Classification Approaches 13.3 Evolving the Architecture and Parameters of DNNs for Malware Detection 13.3.1 Genetic Algorithm Description 13.3.2 Genetic Operators 13.3.3 Finite State Machine 13.3.4 Fitness Function 13.4 Experimental Setup 13.4.1 Dataset 13.4.2 Algorithm Parametrisation 13.4.3 Experimental Environment 13.5 Results 13.6 Conclusions References 14 Data Dieting in GAN Training 14.1 Introduction 14.2 General GAN Training 14.3 Related Work 14.4 Data Reduction in Evolutionary GAN Training 14.4.1 Overview of Redux-Lipizzaner 14.4.2 Dataset Sampling in Redux-Lipizzaner 14.4.3 Evolving Generator Mixture Weights 14.4.4 Algorithms of Redux-Lipizzaner 14.5 Experimental Analysis 14.5.1 Experimental Setup 14.5.2 Research Question 1: How Does the Accuracy of Generators Change in Spatially Distributed Grids When the Dataset Size Is Decreased? 14.5.3 Research Question 2: Given We Use Ensembles, If We Reduce the Data Quantity at Each Cell, at What Point Will the Ensemble Fail to Unify the Resulting Models Towards Achieving High Accuracy? 14.6 Conclusions and Future Work References 15 One-Pixel Attack: Understanding and Improving Deep Neural Networks with Evolutionary Computation 15.1 Introduction 15.2 Adversarial Machine Learning: A Brief Introduction 15.2.1 The Constraint 15.3 One-Pixel Attack 15.3.1 How Is It Possible? 15.3.2 Are Deep Neural Networks Chaotic Systems? 15.4 Do Deep Neural Networks Learn High-Level Features? 15.4.1 Propagation Maps 15.4.2 Texture-Based Features 15.4.3 Non-robust Features Are Enough 15.5 Robustness vs. Accuracy: Different Objectives 15.6 Attacks on Deep Neural Networks 15.6.1 Are These Attacks Feasible in the Real World? 15.7 Defense Systems 15.7.1 Detection Systems 15.8 Overview of the Current State of Robustness: Evaluating Algorithms and Defenses 15.8.1 Threshold Attack (L∞ Black-Box Attack) 15.8.2 Few-Pixel Attack (L0 Black-Box Attack) 15.8.3 Analysis and Discussion 15.9 Down the Rabbit Hole: The Representation Problem 15.9.1 Representation Metric 15.9.1.1 Davies–Bouldin Metric: Clustering Hypothesis 15.9.2 Analysis 15.10 The Role of Evolutionary Computation 15.10.1 Evolving Robust Architectures 15.10.2 Neuroevolution 15.10.3 Self-Organizing Classifiers 15.10.4 Hybrids 15.11 What Lies Beyond References Index