Stochastic Approximation and Recursive Algorithms and Applications (Stochastic Modelling and Applied Probability (35))

The book presents a thorough development of the modern theory of stochastic approximation or recursive stochastic algorithms for both constrained and unconstrained problems. There is a complete development of both probability one and weak convergence methods for very general noise processes. The proofs of convergence use the ODE method, the most powerful to date, with which the asymptotic behavior is characterized by the limit behavior of a mean ODE. The assumptions and proof methods are designed to cover the needs of recent applications. The development proceeds from simple to complex problems, allowing the underlying ideas to be more easily understood. Rate of convergence, iterate averaging, high-dimensional problems, stability-ODE methods, two time scale, asynchronous and decentralized algorithms, general correlated and state-dependent noise, perturbed test function methods, and large devitations methods, are covered. Many motivational examples from learning theory, ergodic cost problems for discrete event systems, wireless communications, adaptive control, signal processing, and elsewhere, illustrate the application of the theory.; This second edition is a thorough revision, although the main features and the structure remain unchanged. It contains many additional applications and results, and more detailed discussion. Harold J. Kushner is a University Professor and Professor of Applied Mathematics at Brown University. He has written numerous books and articles on virtually all aspects of stochastic systems theory, and has received various awards including the IEEE Control Systems Field Award The basic stochastic approximation algorithms introduced by Robbins and MonroandbyKieferandWolfowitzintheearly1950shavebeenthesubject of an enormous literature, both theoretical and applied. This is due to the large number of applications and the interesting theoretical issues in the analysis of “dynamically de?ned” stochastic processes. The basic paradigm is a stochastic di?erence equation such as? =? + Y, where? takes n+1 n n n n its values in some Euclidean space, Y is a random variable, and the “step n size” > 0 is small and might go to zero as n??. In its simplest form, n? is a parameter of a system, and the random vector Y is a function of n “noise-corrupted” observations taken on the system when the parameter is set to?. One recursively adjusts the parameter so that some goal is met n asymptotically. Thisbookisconcernedwiththequalitativeandasymptotic properties of such recursive algorithms in the diverse forms in which they arise in applications. There are analogous continuous time algorithms, but the conditions and proofs are generally very close to those for the discrete time case. The original work was motivated by the problem of?nding a root of a continuous function g ̄(?), where the function is not known but the - perimenter is able to take “noisy” measurements at any desired value of?. Recursive methods for root?nding are common in classical numerical analysis, and it is reasonable to expect that appropriate stochastic analogs would also perform well.

This revised and expanded second edition presents a thorough development of the modern theory of stochastic approximation or recursive stochastic algorithms for both constrained and unconstrained problems. There is a complete development of both probability one and weak convergence methods for very general noise processes. The proofs of convergence use the ODE method, the most powerful to date. The assumptions and proof methods are designed to cover the needs of recent applications. The development proceeds from simple to complex problems, allowing the underlying ideas to be more easily understood. Rate of convergence, iterate averaging, high-dimensional problems, stability-ODE methods, two time scale, asynchronous and decentralized algorithms, state-dependent noise, stability methods for correlated noise, perturbed test function methods, and large deviations methods are covered. Many motivating examples from learning theory, ergodic cost problems for discrete event systems, wireless communications, adaptive control, signal processing, and elsewhere illustrate the applications of the theory.

The book presents a comprehensive development of the modern theory of stochastic approximation, or recursive stochastic algorithms, for both constrained and unconstrained problems, with step sizes that either go to zero or are constant and small (and perhaps random). The general motivation arises from the new challenges in applications that have arisen in recent years. There is a thorough treatment of both probability one and weak convergence methods for very general noise models. The convergence proofs are built around the powerful ODE (ordinary, differential equation) method, which characterizes the limit behavior of the algorithm in terms of the asymptotics of a "mean limit ODE" or an analogous dynamical system. Not only is the method particularly convenient for dealing with complicated noise and dynamics, but also greatly simplifies the treatment of the more classical cases Front Matter....Pages i-xxi Introduction: Applications and Issues....Pages 1-24 Applications to Learning, State Dependent Noise, and Queueing....Pages 25-46 Applications in Signal Processing and Adaptive Control....Pages 47-66 Mathematical Background....Pages 67-83 Convergence with Probability One: Martingale Difference Noise....Pages 85-133 Convergence with Probability One: Correlated Noise....Pages 135-184 Weak Convergence: Introduction....Pages 185-212 Weak Convergence Methods for General Algorithms....Pages 213-250 Applications: Proofs of Convergence....Pages 251-272 Rate of Convergence....Pages 273-325 Averaging of the Iterates....Pages 327-346 Distributed/Decentralized and Asynchronous Algorithms....Pages 347-391 Back Matter....Pages 393-417 In recent years algorithms of the stochastic approximation type have found applications in new and diverse areas, and new techniques have been developed for proofs of convergence and rate of convergence. The actual and potential applications in signal processing have exploded. New challenges have arisen in applications to adaptive control. This book presents a thorough coverage of the ODE method used to analyze these algorithms. This book presents a thorough development of the modern theory of stochastic approximation or recursive stochastic algorithms for both constrained and unconstrained problems. This second edition is a thorough revision, although the main features and structure remain unchanged. It contains many additional applications and results as well as more detailed discussion.-- Provided by Publisher There is a thorough treatment of rate of convergence, iterate averaging, high-dimensional problems, ergodic cost problems, stability methods for correlated noise, and decentralized and asynchronous algorithms