Markov Processes and Controlled Markov Chains

The general theory of stochastic processes and the more specialized theory of Markov processes evolved enormously in the second half of the last century. In parallel, the theory of controlled Markov chains (or Markov decision processes) was being pioneered by control engineers and operations researchers. Researchers in Markov processes and controlled Markov chains have been, for a long time, aware of the synergies between these two subject areas. However, this may be the first volume dedicated to highlighting these synergies and, almost certainly, it is the first volume that emphasizes the contributions of the vibrant and growing Chinese school of probability. The chapters that appear in this book reflect both the maturity and the vitality of modern day Markov processes and controlled Markov chains. They also will provide an opportunity to trace the connections that have emerged between the work done by members of the Chinese school of probability and the work done by the European, US, Central and South American and Asian scholars. Front Matter....Pages i-x Front Matter....Pages 1-1 Branching Exit Markov System and their Applications to Partial Differential Equations....Pages 3-13 Feller Transition Functions, Resolvent Decomposition Theorems, and their Application in Unstable Denumerable Markov Processes....Pages 15-39 Identifying Q -Processes with a Given Finite μ -Invariant Measure....Pages 41-55 Convergence Property of Standard Transition Functions....Pages 57-67 Markov Skeleton Processes....Pages 69-92 Piecewise Deterministic Markov Processes and Semi-Dynamic Systems....Pages 93-107 Front Matter....Pages 109-109 Average Optimality for Adaptive Markov Control Processes with Unbounded Costs and Unknown Disturbance Distribution....Pages 111-134 Controlled Markov Chains with Utility Functions....Pages 135-149 Classification Problems in MDPs....Pages 151-165 Optimality Conditions for CTMDP with Average Cost Criterion....Pages 167-188 Optimal and Nearly Optimal Policies in Markov Decision Chains with Nonnegative Rewards and Risk-Sensitive Expected Total-Reward Criterion....Pages 189-221 Interval Methods for Uncertain Markov Decision Processes....Pages 223-232 Constrained Discounted Semi-Markov Decision Processes....Pages 233-244 Linear Program for Communicating MDPs with Multiple Constraints....Pages 245-254 Optimal Switching Problem for Markov Chains....Pages 255-286 Approximations of a Controlled Diffusion Model for Renewable Resource Exploitation....Pages 287-302 Front Matter....Pages 303-303 A Fleming-Viot Process with Unbounded Selection, II....Pages 305-322 Boundary Theory for Superdiffusions....Pages 323-330 On Solutions of Backward Stochastic Differential Equations with Jumps and Stochastic Control....Pages 331-340 Doob’s Inequality and Lower Estimation of The Maximum of Martingales....Pages 341-349 Front Matter....Pages 303-303 The Hausdorff Measure of the Level Sets of Brownian Motion on the Sierpinski Carpet....Pages 351-361 Monotonic Approximation of the Gittins Index....Pages 363-367 Front Matter....Pages 369-369 Optimal Consumption-Investment Decisions Allowing for Bankruptcy: A Brief Survey....Pages 371-387 The Hedging Strategy of an Asian Option....Pages 389-395 The Pricing of Options to Exchange One Asset for Another....Pages 397-404 Finite Horizon Portfolio Risk Models with Probability Criterion....Pages 405-424 Long Term Average Control of a Local Time Process....Pages 425-441 Singularly Perturbed Hybrid Control Systems Approximated by Structured Linear Programs....Pages 443-463 The Effect of Stochastic Disturbance on the Solitary Waves....Pages 465-474 Independent Candidate for Tierney Model of H-M Algorithms....Pages 475-487 How Rates of Convergence for Gibbs Fields Depend on the Interaction and the Kind of Scanning Used....Pages 489-498 Expected Loss and Availability of Multistate Repairable System....Pages 499-511