Decision Processes in Dynamic Probabilistic Systems [Paperback]

1 Semi-Markov and Markov Chains.- 1.1 Definitions and basic properties.- 1.1.1 Discrete time semi-Markov and Markov behaviour of systems.- 1.1.2 Multi-step transition probability.- 1.1.3 Semi-Markov processes.- 1.1.4 State occupancy and waiting-time statistics.- 1.1.5 Non-homogeneous Markov processes.- 1.1.6 The limit theorem.- 1.1.7 Effect of small deviations in the transition probability matrix.- 1.1.7.1 Limits of some important characteristics for Markov chains.- 1.2 Algebraic and analytical methods in the study of Markovian systems.- 1.2.1 Eigenvalues and eigenvectors.- 1.2.2 Stochastic matrices.- 1.2.3 Perron-Frobenius theorem.- 1.2.4 The Geometric transformation (the z-transform).- 1.2.5 Exponential transformation (Laplace transform).- 1.3 Transient and recurrent processes.- 1.3.1 Transient processes.- 1.3.2 The study of recurrent state occupancy in Markov processes.- 1.4 Markovian populations.- 1.4.1 Vectorial processes with a Markovian structure.- 1.4.2 General branching processes.- 1.5 Partially observable Markov chains.- 1.5.1 The core process.- 1.5.2 The observation process.- 1.5.3 The state of knowledge and its dynamics.- 1.5.4 Examples.- 1.6 Rewards and discounting.- 1.6.1 Rewards for sequential decision processes.- 1.6.2 Rewards in decision processes with Markov structure.- 1.6.3 Markovian decision processes with and without discounting.- 1.7 Models and applications.- 1.7.1 Real systems with a Markovian structure.- 1.7.2 Model formulation and practical results.- 1.7.2.1 A semi-Markov model for hospital planning.- 1.7.2.2 System reliability.- 1.7.2.3 A Markovian interpretation for PERT networks.- 1.8 Dynamic-decision models for clinical diagnosis.- 1.8.1 Pattern recognition.- 1.8.2 Model optimization.- 2 Dynamic and Linear Programming.- 2.1 Discrete dynamic programming.- 2.2 A linear programming formulation and an algorithm for computation.- 2.2.1 A general formulation for the LP problem and the Simplex method.- 2.2.2 Linear programming - a matrix formuló.