Practical Neural Network Recipies in C++

this Text Serves As A Cookbook For Neural Network Solutions To Practical Problems Using C++. It Will Enable Those With Moderate Programming Experience To Select A Neural Network Model Appropriate To Solving A Particular Problem, And To Produce A Working Program Implementing That Network. The Book Provides Guidance Along The Entire Problem-solving Path, Including Designing The Training Set, Preprocessing Variables, Training And Validating The Network, And Evaluating Its Performance. Though The Book Is Not Intended As A General Course In Neural Networks, No Background In Neural Works Is Assumed And All Models Are Presented From The Ground Up. the Principle Focus Of The Book Is The Three Layer Feedforward Network, For More Than A Decade As The Workhorse Of Professional Arsenals. Other Network Models With Strong Performance Records Are Also Included. bound In The Book Is An Ibm Diskette That Includes The Source Code For All Programs In The Book. Much Of This Code Can Be Easily Adapted To C Compilers. In Addition, The Operation Of All Programs Is Thoroughly Discussed Both In The Text And In The Comments Within The Code To Facilitate Translation To Other Languages. booknews for Those With Moderate Programming Experience, This Text/software Package (5.25 Dos Format) Is A Recipe Book Of C++ Programs Adaptable To Problems Requiring A Neural Network Approach. Annotation C. Book News, Inc., Portland, Or (booknews.com) Contents......Page all_10073_to_00502.cpc0006.djvu Preface......Page all_10073_to_00502.cpc0013.djvu 1. Foundations......Page all_10073_to_00502.cpc0015.djvu Motivation......Page all_10073_to_00502.cpc0016.djvu New Life for Old Techniques......Page all_10073_to_00502.cpc0017.djvu Perceptrons and Linear Separability......Page all_10073_to_00502.cpc0018.djvu Neural Network Capabilities......Page all_10073_to_00502.cpc0020.djvu Basic Structure of a Neural Network......Page all_10073_to_00502.cpc0022.djvu Training......Page all_10073_to_00502.cpc0023.djvu Validation......Page all_10073_to_00502.cpc0024.djvu Leave-k-out Method......Page all_10073_to_00502.cpc0026.djvu 2. Classification......Page all_10073_to_00502.cpc0029.djvu Binary Decisions......Page all_10073_to_00502.cpc0030.djvu Making the Decision......Page all_10073_to_00502.cpc0031.djvu Reject Category......Page all_10073_to_00502.cpc0032.djvu Other Encoding Schemes......Page all_10073_to_00502.cpc0033.djvu Supervised versus Unsupervised Training......Page all_10073_to_00502.cpc0035.djvu 3. Autoassociation......Page all_10073_to_00502.cpc0037.djvu Autoassociative Filtering......Page all_10073_to_00502.cpc0038.djvu Code for Autoassociative Filtering......Page all_10073_to_00502.cpc0042.djvu Noise Reduction......Page all_10073_to_00502.cpc0043.djvu Learning a Prototype from Exemplars......Page all_10073_to_00502.cpc0045.djvu Exposing Isolated Events......Page all_10073_to_00502.cpc0046.djvu Pattern Completion......Page all_10073_to_00502.cpc0054.djvu Error Correction......Page all_10073_to_00502.cpc0055.djvu Encoding Words......Page all_10073_to_00502.cpc0056.djvu Data Compression......Page all_10073_to_00502.cpc0058.djvu 4. Time-Series Prediction......Page all_10073_to_00502.cpc0060.djvu The Basic Model......Page all_10073_to_00502.cpc0062.djvu Input Data......Page all_10073_to_00502.cpc0063.djvu Trend Elimination......Page all_10073_to_00502.cpc0064.djvu Code for Detrending and Retrending......Page all_10073_to_00502.cpc0068.djvu Seasonal Variation......Page all_10073_to_00502.cpc0071.djvu Scaling......Page all_10073_to_00502.cpc0073.djvu Multiple Prediction......Page all_10073_to_00502.cpc0074.djvu Multiple Predictors......Page all_10073_to_00502.cpc0075.djvu Measuring Prediction Error......Page all_10073_to_00502.cpc0077.djvu 5. Function Approximation......Page all_10073_to_00502.cpc0080.djvu Univariate Function Approximation......Page all_10073_to_00502.cpc0081.djvu Inverse Modeling......Page all_10073_to_00502.cpc0085.djvu Multiple Regression......Page all_10073_to_00502.cpc0087.djvu 6. Multilayer Feedforward Networks......Page all_10073_to_00502.cpc0090.djvu Basic Architecture......Page all_10073_to_00502.cpc0091.djvu Activation Functions......Page all_10073_to_00502.cpc0093.djvu Example Network......Page all_10073_to_00502.cpc0095.djvu Linear Output Neurons......Page all_10073_to_00502.cpc0097.djvu Theoretical Discussion......Page all_10073_to_00502.cpc0098.djvu Bibliography of Feedforward Network Theory......Page all_10073_to_00502.cpc0101.djvu Algorithms for Executing the Network......Page all_10073_to_00502.cpc0103.djvu Training the Network......Page all_10073_to_00502.cpc0107.djvu Training by Backpropagation of Errors......Page all_10073_to_00502.cpc0113.djvu Training by Conjugate Gradients......Page all_10073_to_00502.cpc0118.djvu Minimizing along a Direction......Page all_10073_to_00502.cpc0119.djvu Choosing the Direction for Minimization......Page all_10073_to_00502.cpc0123.djvu Eluding Local Minima in Learning......Page all_10073_to_00502.cpc0124.djvu Local Minima Happen Easily......Page all_10073_to_00502.cpc0125.djvu Mistaken Minima......Page all_10073_to_00502.cpc0127.djvu Other Means of Escape......Page all_10073_to_00502.cpc0128.djvu When to Use a Multiple-Layer Feedforward Network......Page all_10073_to_00502.cpc0129.djvu 7. Eluding Local Minima I: Simulated Annealing......Page all_10073_to_00502.cpc0130.djvu Overview......Page all_10073_to_00502.cpc0131.djvu Choosing the Annealing Parameters......Page all_10073_to_00502.cpc0132.djvu Implementation in Feedforward Network Learning......Page all_10073_to_00502.cpc0134.djvu A Sample Program......Page all_10073_to_00502.cpc0135.djvu A Sample Function......Page all_10073_to_00502.cpc0139.djvu Random Number Generation......Page all_10073_to_00502.cpc0141.djvu Going on from Here......Page all_10073_to_00502.cpc0145.djvu 8. Eluding Local Minima II: Genetic Optimization......Page all_10073_to_00502.cpc0148.djvu Overview......Page all_10073_to_00502.cpc0149.djvu Designing the Genetic Structure......Page all_10073_to_00502.cpc0151.djvu Evaluation......Page all_10073_to_00502.cpc0153.djvu Parent Selection......Page all_10073_to_00502.cpc0157.djvu Reproduction......Page all_10073_to_00502.cpc0160.djvu Mutation......Page all_10073_to_00502.cpc0161.djvu A Genetic Minimization Subroutine......Page all_10073_to_00502.cpc0162.djvu Some Functions for Genetic Optimization......Page all_10073_to_00502.cpc0168.djvu Gray Codes......Page all_10073_to_00502.cpc0170.djvu Two-Point Crossover......Page all_10073_to_00502.cpc0172.djvu 9. Regression and Neural Networks......Page all_10073_to_00502.cpc0178.djvu Overview......Page all_10073_to_00502.cpc0179.djvu Singular-Value Decomposition......Page all_10073_to_00502.cpc0180.djvu Regression in Neural Networks......Page all_10073_to_00502.cpc0182.djvu 10. Designing Feedforward Network Architectures......Page all_10073_to_00502.cpc0186.djvu How Many Hidden Layers?......Page all_10073_to_00502.cpc0187.djvu How Many Hidden Neurons?......Page all_10073_to_00502.cpc0189.djvu How Long Do I Train This Thing???......Page all_10073_to_00502.cpc0193.djvu 11. Interpreting Weights: How Does This Thing Work?......Page all_10073_to_00502.cpc0199.djvu Features Used by Networks in General......Page all_10073_to_00502.cpc0202.djvu Examination of Weight Vectors......Page all_10073_to_00502.cpc0203.djvu Hinton Diagrams......Page all_10073_to_00502.cpc0204.djvu Clustering......Page all_10073_to_00502.cpc0206.djvu Sensitivity Analysis......Page all_10073_to_00502.cpc0207.djvu Stereotypical Inputs......Page all_10073_to_00502.cpc0209.djvu 12. Probabilistic Neural Networks......Page all_10073_to_00502.cpc0213.djvu Overview......Page all_10073_to_00502.cpc0214.djvu Computational Aspects......Page all_10073_to_00502.cpc0220.djvu Optimizing Sigma......Page all_10073_to_00502.cpc0221.djvu Related Models......Page all_10073_to_00502.cpc0222.djvu A Sample Program......Page all_10073_to_00502.cpc0223.djvu Optimizing Sigma......Page all_10073_to_00502.cpc0225.djvu Other Optimization Criteria......Page all_10073_to_00502.cpc0230.djvu Bayesian Confidence Measures......Page all_10073_to_00502.cpc0231.djvu Autoassociative Versions......Page all_10073_to_00502.cpc0232.djvu When to Use a Probabilistic Neural Network......Page all_10073_to_00502.cpc0233.djvu 13. Functional Link Networks......Page all_10073_to_00502.cpc0235.djvu Application to Nonlinear Approximation......Page all_10073_to_00502.cpc0238.djvu Mathematics of the Functional Link Network......Page all_10073_to_00502.cpc0239.djvu When to Use a Functional Link Network......Page all_10073_to_00502.cpc0241.djvu 14. Hybrid Networks......Page all_10073_to_00502.cpc0243.djvu Functional Link Net as a Hidden Layer......Page all_10073_to_00502.cpc0244.djvu Fast Bayesian Confidences......Page all_10073_to_00502.cpc0247.djvu Training......Page all_10073_to_00502.cpc0250.djvu Attention-based Processing......Page all_10073_to_00502.cpc0251.djvu Factorable Problems......Page all_10073_to_00502.cpc0254.djvu Training the Data Reduction Networks......Page all_10073_to_00502.cpc0255.djvu Splitting Is Not Always Effective......Page all_10073_to_00502.cpc0256.djvu 15. Designing the Training Set......Page all_10073_to_00502.cpc0257.djvu Number of Samples......Page all_10073_to_00502.cpc0258.djvu Overfitting......Page all_10073_to_00502.cpc0259.djvu Network Size Affects Training Set Size......Page all_10073_to_00502.cpc0260.djvu Borderline Cases......Page all_10073_to_00502.cpc0261.djvu Hidden Bias......Page all_10073_to_00502.cpc0262.djvu Fudging Cases......Page all_10073_to_00502.cpc0263.djvu 16. Preparing Input Data......Page all_10073_to_00502.cpc0265.djvu General Considerations......Page all_10073_to_00502.cpc0266.djvu Nominal Variables......Page all_10073_to_00502.cpc0267.djvu Ordinal Variables......Page all_10073_to_00502.cpc0271.djvu Interval Variables......Page all_10073_to_00502.cpc0274.djvu Is Scaling Always Necessary?......Page all_10073_to_00502.cpc0278.djvu Transformations......Page all_10073_to_00502.cpc0279.djvu Circular Discontinuity......Page all_10073_to_00502.cpc0282.djvu View Angles......Page all_10073_to_00502.cpc0283.djvu Hue......Page all_10073_to_00502.cpc0284.djvu Outliers......Page all_10073_to_00502.cpc0286.djvu Discarding Data......Page all_10073_to_00502.cpc0287.djvu Missing Data......Page all_10073_to_00502.cpc0288.djvu 17. Fuzzy Data and Processing......Page all_10073_to_00502.cpc0291.djvu Treating Fuzzy Values as Nominal and Ordinal......Page all_10073_to_00502.cpc0293.djvu Advantages of Fuzzy Set Processing......Page all_10073_to_00502.cpc0294.djvu The Neural Network - Fuzzy Set Interface......Page all_10073_to_00502.cpc0295.djvu Membership Functions......Page all_10073_to_00502.cpc0296.djvu Continuous Variables......Page all_10073_to_00502.cpc0299.djvu Multivariate Domains......Page all_10073_to_00502.cpc0300.djvu Hedges......Page all_10073_to_00502.cpc0301.djvu Negation, Conjunction, and Disjunction......Page all_10073_to_00502.cpc0302.djvu Modus Ponens......Page all_10073_to_00502.cpc0304.djvu Combining Operations......Page all_10073_to_00502.cpc0307.djvu Defuzzification......Page all_10073_to_00502.cpc0311.djvu Maximum Height Method......Page all_10073_to_00502.cpc0312.djvu Centroid Method......Page all_10073_to_00502.cpc0313.djvu Constructors......Page all_10073_to_00502.cpc0315.djvu Negation and Scaling......Page all_10073_to_00502.cpc0319.djvu Conjunction and Disjunction......Page all_10073_to_00502.cpc0320.djvu Centroid......Page all_10073_to_00502.cpc0326.djvu Simplifying Interactions......Page all_10073_to_00502.cpc0328.djvu Fuzzy One-of-n Coding......Page all_10073_to_00502.cpc0329.djvu Simple Membership Output......Page all_10073_to_00502.cpc0331.djvu Postprocessing with Defuzzification......Page all_10073_to_00502.cpc0332.djvu 18. Unsupervised Training......Page all_10073_to_00502.cpc0339.djvu Input Normalization......Page all_10073_to_00502.cpc0342.djvu Z-Axis Normalization......Page all_10073_to_00502.cpc0343.djvu Training the Kohonen Network......Page all_10073_to_00502.cpc0344.djvu Updating the Weights......Page all_10073_to_00502.cpc0346.djvu Learning Rate......Page all_10073_to_00502.cpc0348.djvu Measuring Network Error......Page all_10073_to_00502.cpc0349.djvu Determining Convergence......Page all_10073_to_00502.cpc0350.djvu Neurons That Refuse to Learn......Page all_10073_to_00502.cpc0351.djvu Self-Organization......Page all_10073_to_00502.cpc0352.djvu 19. Evaluating Performance of Neural Networks......Page all_10073_to_00502.cpc0354.djvu Mean Square Error......Page all_10073_to_00502.cpc0355.djvu Problems with Mean Square Error......Page all_10073_to_00502.cpc0356.djvu Relatives of Mean Square Error......Page all_10073_to_00502.cpc0357.djvu Cost Functions......Page all_10073_to_00502.cpc0358.djvu Confusion Matrix......Page all_10073_to_00502.cpc0359.djvu ROC (Receiver Operating Characteristic) Curves......Page all_10073_to_00502.cpc0362.djvu Computing the ROC Curve Area......Page all_10073_to_00502.cpc0365.djvu Cost Functions and ROC Curves......Page all_10073_to_00502.cpc0368.djvu Signal-to-Noise Ratio......Page all_10073_to_00502.cpc0370.djvu 20. Confidence Measures......Page all_10073_to_00502.cpc0372.djvu Testing Individual Hypotheses......Page all_10073_to_00502.cpc0373.djvu Computing Confidence......Page all_10073_to_00502.cpc0378.djvu Confidence in the Null Hypothesis......Page all_10073_to_00502.cpc0379.djvu Multiple Classes......Page all_10073_to_00502.cpc0380.djvu Confidence in the Confidence......Page all_10073_to_00502.cpc0381.djvu Example Programs......Page all_10073_to_00502.cpc0382.djvu Sorting......Page all_10073_to_00502.cpc0383.djvu Estimating the Distribution......Page all_10073_to_00502.cpc0384.djvu Estimating Confidences......Page all_10073_to_00502.cpc0385.djvu Bayesian Methods......Page all_10073_to_00502.cpc0387.djvu Example Program......Page all_10073_to_00502.cpc0392.djvu Multiple Classes......Page all_10073_to_00502.cpc0393.djvu Hypothesis Testing versus Bayes' Method......Page all_10073_to_00502.cpc0395.djvu 21. Optimizing the Decision Threshold......Page all_10073_to_00502.cpc0400.djvu 22. Using the NEURAL Program......Page all_10073_to_00502.cpc0413.djvu GENERAL Model......Page all_10073_to_00502.cpc0415.djvu The LAYER Network Model......Page all_10073_to_00502.cpc0416.djvu Initialization by Genetic Optimization......Page all_10073_to_00502.cpc0417.djvu Learning......Page all_10073_to_00502.cpc0418.djvu The KOHONEN Network Model......Page all_10073_to_00502.cpc0419.djvu Initialization and Learning......Page all_10073_to_00502.cpc0420.djvu Saving Weights and Execution Results......Page all_10073_to_00502.cpc0422.djvu Alphabetical Glossary of Commands......Page all_10073_to_00502.cpc0423.djvu Verification of Program Operation......Page all_10073_to_00502.cpc0427.djvu Appendix......Page all_10073_to_00502.cpc0433.djvu Bibliography......Page all_10073_to_00502.cpc0488.djvu Index......Page all_10073_to_00502.cpc0500.djvu This text serves as a cookbook for neural network solutions to practical problems using C++. It will enable those with moderate programming experience to select a neural network model appropriate to solving a particular problem, and to produce a working program implementing that network. The book provides guidance along the entire problem-solving path, including designing the training set, preprocessing variables, training and validating the network, and evaluating its performance. Though the book is not intended as a general course in neural networks, no background in neural works is assumed and all models are presented from the ground up. The principle focus of the book is the three layer feedforward network, for more than a decade as the workhorse of professional arsenals. Other network models with strong performance records are also included. Bound in the book is an IBM diskette that includes the source code for all programs in the book. Much of this code can be easily adapted to C compilers. In addition, the operation of all programs is thoroughly discussed both in the text and in the comments within the code to facilitate translation to other languages.