## The ODE method and spectral theory of Markov operators (2002)

Venue: | IN STOCHASTIC THEORY AND CONTROL (LAWRENCE, KS, 2001), SER. LECTURE NOTES IN CONTROL AND INFORMATION SCIENCES |

Citations: | 10 - 2 self |

### BibTeX

@INPROCEEDINGS{Huang02theode,

author = {Jianyi Huang and Ioannis Kontoyiannis and Sean P. Meyn},

title = {The ODE method and spectral theory of Markov operators},

booktitle = {IN STOCHASTIC THEORY AND CONTROL (LAWRENCE, KS, 2001), SER. LECTURE NOTES IN CONTROL AND INFORMATION SCIENCES},

year = {2002},

pages = {205--221},

publisher = {Springer}

}

### Years of Citing Articles

### OpenURL

### Abstract

We give a development of the ODE method for the analysis of recursive algorithms described by a stochastic recursion. With variability modeled via an underlying Markov process, and under general assumptions, the following results are obtained: (i) Stability of an associated ODE implies that the stochastic recursion is stable in a strong sense when a gain parameter is small. (ii) The range of gain-values is quantified through a spectral analysis of an associated linear operator, providing a non-local theory, even for nonlinear systems. (iii) A second-order analysis shows precisely how variability leads to sensitivity of the algorithm with respect to the gain parameter. All results are obtained within the natural operator-theoretic framework of geometrically ergodic Markov processes.

### Citations

390 |
Stochastic Approximation Algorithms and Applications
- Kushner, Yin
- 1997
(Show Context)
Citation Context ...that, if the ODE (8) is stable, then the stochastic model (1) is stable in a statistical sense for a range of small , and comparisons with (9) are possible under still stronger assumptions (see e.g. [=-=4,8,21,20,14-=-] for results concerning both linear and nonlinear recursions). In [27] an alternative point of view was proposed where the stability verication problem for (1) is cast in terms of the spectral radius... |

364 |
Dynamical Systems V
- Arnol’d
- 1994
(Show Context)
Citation Context ...are of interest in a wide range ofselds. Application areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems =-=[1]-=-, queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [3... |

359 |
Adaptive Algorithms and Stochastic Approximations
- Benveniste, Métivier, et al.
- 1990
(Show Context)
Citation Context ...avior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms =-=[11,5,27,17]-=-, perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigen... |

333 |
Non-negative Matrices and Markov Chains
- Seneta
- 1981
(Show Context)
Citation Context ...envalue of L for a range of 0, and we use this fact to obtain a multiplicative ergodic theorem. The maximal eigenvalue in Theorem 3 is a generalization of the PerronFrobenius eigenvalue; c.f. [31,1=-=8]. Th-=-eorem 3. Suppose that the eigenvalues f i (M)g of M are distinct, then 8 J. Huang, I. Kontoyiannis, S.P. Meyn (i) There exists " 0 > 0;s0 > 0 such that the linear operator L z has exactly k disti... |

152 |
Approximation and weak convergence methods for random processes, with applications to stochastic systems theory
- Kushner
- 1984
(Show Context)
Citation Context ...that, if the ODE (8) is stable, then the stochastic model (1) is stable in a statistical sense for a range of small , and comparisons with (9) are possible under still stronger assumptions (see e.g. [=-=4,8,21,20,14-=-] for results concerning both linear and nonlinear recursions). In [27] an alternative point of view was proposed where the stability verication problem for (1) is cast in terms of the spectral radius... |

115 |
Products of random matrices
- Furstenberg, Kesten
- 1960
(Show Context)
Citation Context ...sis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in =-=[3,13,29,28]-=-. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [33,16] for a recent application to capacity of communication channels). Although the results... |

62 | The O.D.E. method for convergence of stochastic approximation and reinforcement learning
- Borkar, Meyn
(Show Context)
Citation Context ...that, if the ODE (8) is stable, then the stochastic model (1) is stable in a statistical sense for a range of small , and comparisons with (9) are possible under still stronger assumptions (see e.g. [=-=4,8,21,20,14-=-] for results concerning both linear and nonlinear recursions). In [27] an alternative point of view was proposed where the stability verication problem for (1) is cast in terms of the spectral radius... |

50 | Computable Bounds for Geometric Convergence Rates of Markov Chains
- Meyn, Tweedie
- 1994
(Show Context)
Citation Context ...es on both sides of the eigenfunction equation for Q gives, Q 0 h +Qh 0 = 0 h + h 0 (25) Setting = 0 gives a version of Poisson's equation, Q 0 0 h 0 +Qh 0 0 = 0 0 h 0 + 0 h 0 0 (26) Using the identities of h 0 and Q 0 0 h 0 = E x [ M T 1 h 0 h 0 M 1 ], we obtain the steady state expression M T h 0 + h 0 M = 0 0 h 0 : (27) Since M = I , we have 0 0 = 2. Now, taking the 2nd de... |

47 |
Dynamics of stochastic approximation algorithms, Séminaire de
- Benäım
- 1999
(Show Context)
Citation Context ...that, if the ODE (8) is stable, then the stochastic model (1) is stable in a statistical sense for a range of small α, and comparisons with (9) are possible under still stronger assumptions (see e.g. =-=[4,8,21,20,14]-=- for results concerning both linear and nonlinear recursions). In [27] an alternative point of view was proposed where the stability verification problem for (1) is cast in terms of the spectral radiu... |

46 |
A multiplicative ergodic theorem. Characteristic Ljapunov, exponents of dynamical systems
- Oseledec
- 1968
(Show Context)
Citation Context ...sis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in =-=[3,13,29,28]-=-. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [33,16] for a recent application to capacity of communication channels). Although the results... |

44 |
Coupling, Stationarity and Regeneration
- Thörisson
- 2000
(Show Context)
Citation Context ...s shows that E x [kX t k 2 ] is bounded in t 0 for any deterministic initial conditions 0 = x 2 X, X 0 =s2 R k . To construct the stationary process X we apply backward coupling as presented in [32]. Consider the system starting at time n, initialized ats= 0, and let X ;n t , t n, denote the resulting state trajectory. We then have for all n; m 1, X ;m t X ;n t = 0 Y i=t (I M i ) [X ... |

44 |
Multiaccess fading channels. I. polymatroid structure, optimal resource allocation and throughput capacities
- Tse, Hanly
- 1998
(Show Context)
Citation Context ...1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. =-=[33,1-=-6] for a recent application to capacity of communication channels). Although the results of the present paper do not address these issues, they provide justication for simplied models in communication... |

38 | Spectral theory and limit theorems for geometrically ergodic Markov processes
- Kontoyiannis, Meyn
- 2003
(Show Context)
Citation Context ...roup of linear operators. This approach is based on the functional analytic setting of [26], and analogous techniques are used in the treatment of multiplicative ergodic theory and spectral theory in =-=[2,18,1-=-9]. The main results of [27] may be interpreted as a signicant extension of the ODE method for linear recursions. Our present results give a unied treatment of both the linear and nonlinear models tre... |

26 | Random Fibonacci sequences and the number 1.13198824
- Viswanath
(Show Context)
Citation Context ...oducts Q s i=t (I M i ) play an important role in the behavior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [=-=15,34]-=-, statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28... |

22 | Multiplicative ergodicity and large deviations for an irreducible Markov chain. Stochastic Process
- Balaji, Meyn
(Show Context)
Citation Context ...roup of linear operators. This approach is based on the functional analytic setting of [26], and analogous techniques are used in the treatment of multiplicative ergodic theory and spectral theory in =-=[2,18,1-=-9]. The main results of [27] may be interpreted as a signicant extension of the ODE method for linear recursions. Our present results give a unied treatment of both the linear and nonlinear models tre... |

21 | Products of irreducible random matrices in the (max, +) algebra, Adv
- Mairesse
- 1997
(Show Context)
Citation Context ... wide range of fields. Application areas include numerical analysis [12, 30], statistical physics [8, 9], recursive algorithms [10, 23], perturbation theory for dynamical systems [1], queueing theory =-=[19]-=-, and even botany [26]. Seminal results are contained in [3, 25, 24]. 2A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [29, 13] for recent app... |

20 |
On positive real transfer functions and the convergence of some recursive schemes
- Ljung
- 1977
(Show Context)
Citation Context ...X t g: lim t!1 kX t ( ) X t k = 0 : These ideas are related to issues developed in Section 3. The traditional analytic technique for addressing the stability of (6) or of (1) is the ODE method of [2=-=-=-2]. For linear models, the basic idea is that, for small values of , the behavior of (1) should mimic that of the linear ODE, d dtst = Mst +W ; (8) where M and W are steady-state means of M t and W t ... |

15 |
Multiaccess fading channels II: Delay-limited capacities
- Hanly, Tse
- 1998
(Show Context)
Citation Context ...1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. =-=[33,1-=-6] for a recent application to capacity of communication channels). Although the results of the present paper do not address these issues, they provide justication for simplied models in communication... |

12 |
Limit theorems for non-commutative operations
- Bellman
- 1954
(Show Context)
Citation Context ...sis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in =-=[3,13,29,28]-=-. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [33,16] for a recent application to capacity of communication channels). Although the results... |

12 |
Products of random matrices in statistical physics (Springer-Verlag
- Crisanti, Paladin, et al.
- 1993
(Show Context)
Citation Context ... an important role in the behavior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [15,34], statistical physics =-=[9,10]-=-, recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popul... |

11 |
Stochastic approximation algorithms: overview and recent trends
- Bharath, Borkar
- 1999
(Show Context)
Citation Context ...thms and their variants are commonly found in control, communication and relatedselds. Popularity has grown due to increased computing power, and the interest in various `machine learning' algorithms =-=[6,7,12-=-]. When the algorithm is linear, then the error equations take the following linear recursive form, X t+1 = [I M t ] X t +W t+1 ; (1) where X = fX t g is an error sequence, M = fM t g is a sequence of... |

7 | Exponential convergence of products of random matrices, application to the study of adaptive algorithms
- Moustakides
- 1998
(Show Context)
Citation Context ...avior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms =-=[11,5,27,17]-=-, perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigen... |

5 |
Structure theorems for partially asynchronous iterations of a nonnegative matrix with random delays
- Gharavi, Anantharam
- 1999
(Show Context)
Citation Context ...oducts Q s i=t (I M i ) play an important role in the behavior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [=-=15,34]-=-, statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28... |

3 | Products of irreducible random matrices
- Mairesse
- 1997
(Show Context)
Citation Context ...wide range ofselds. Application areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory =-=[23]-=-, and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [33,16] for a recent app... |

2 |
The LMS adaptive algorithm: Asymptotic error analysis
- Dabeer, Masry
- 2000
(Show Context)
Citation Context ...avior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms =-=[11,5,27,17]-=-, perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigen... |

2 |
Almost sure exponential stability of random linear differential equations
- Gerencsér
(Show Context)
Citation Context |

2 |
Markov chains, skew products and ergodic theorems for «general» dynamic systems
- Oseledec
(Show Context)
Citation Context |

2 |
Computable bounds for geometric convergence ratesofMarkovchains. The Annals of Applied Probability
- Meyn, Tweedie
- 1994
(Show Context)
Citation Context ...ivatives on both sides of the eigenfunction equation for Qα gives, Qα ′ hα + Qαh ′ α = η ′ αhα + ηαh ′ α (25) Setting α = 0 gives a version of Poisson’s equation, Q ′ 0h0 + Qh ′ 0 = η ′ 0h0 + η0h ′ 0 =-=(26)-=- Using the identities of h0 and Q ′ 0h0 = Ex[−M T 1 h0 − h0M1], we obtain the steady state expression M T h0 + h0M = −η ′ 0h0. (27) Since M = I, wehaveη ′ 0 = −2. Now, taking the 2nd derivatives on bo... |

1 |
Dynamics of stochastic approximation algorithms
- Benam
- 1999
(Show Context)
Citation Context |

1 |
Neuro-Dynamic Programming. Atena Scienti c
- Bertsekas, Tsitsiklis
- 1996
(Show Context)
Citation Context ...thms and their variants are commonly found in control, communication and relatedselds. Popularity has grown due to increased computing power, and the interest in various `machine learning' algorithms =-=[6,7,12-=-]. When the algorithm is linear, then the error equations take the following linear recursive form, X t+1 = [I M t ] X t +W t+1 ; (1) where X = fX t g is an error sequence, M = fM t g is a sequence of... |

1 |
Limit theorem for products of random matrices with Markovian dependence
- Bougerol
- 1986
(Show Context)
Citation Context ... an important role in the behavior of (1). Properties of products of random matrices are of interest in a wide range ofselds. Application areas include numerical analysis [15,34], statistical physics =-=[9,10]-=-, recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28]. A complementary and popul... |

1 |
Law of the iterated logarithm for a constant-gain linear stochastic gradient algorithm
- Joslin, Heunis
(Show Context)
Citation Context |

1 |
Capacity bene from channel sounding in Rayleigh fading channels
- Medard, Meyn, et al.
(Show Context)
Citation Context ...lts of the present paper do not address these issues, they provide justication for simplied models in communication theory, leading to bounds on the capacity for time-varying communication channels [2=-=4-=-]. The relationship with dynamical systems theory is particularly relevant to the issues addressed here. Consider a nonlinear dynamical system described by the equations, X t+1 = X t f(X t ; t+1 ) +W... |

1 |
M Roerdink. Products of random matrices or "why do biennials live longer than two years? CWI Quarterly
- T
- 1989
(Show Context)
Citation Context ...plication areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany =-=[30]-=-. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [33,16] for a recent application to capacity o... |

1 |
Neuro-Dynamic Programming. Atena Scientific
- Bertsekas, Tsitsiklis
- 1996
(Show Context)
Citation Context ...ms and their variants are commonly found in control, communication and related fields. Popularity has grown due to increased computing power, and the interest in various ‘machine learning’ algorithms =-=[6,7,12]-=-. When the algorithm is linear, then the error equations take the following linear recursive form, Xt+1 =[I − αMt] Xt + Wt+1, (1) where X = {Xt} is an error sequence, M = {Mt} is a sequence of k × k r... |

1 |
Capacity benefits from channel sounding in Rayleigh fading channels
- Medard, Meyn, et al.
(Show Context)
Citation Context ...s of the present paper do not address these issues, they provide justification for simplified models in communication theory, leading to bounds on the capacity for time-varying communication channels =-=[24]-=-. The relationship with dynamical systems theory is particularly relevant to the issues addressed here. Consider a nonlinear dynamical system described by the equations, Xt+1 = Xt − f(Xt,Φt+1)+Wt+1 , ... |

1 |
M Roerdink. Products of random matrices or ”why do biennials live longer than two years
- T
- 1989
(Show Context)
Citation Context ...plication areas include numerical analysis [15,34], statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany =-=[30]-=-. Seminal results are contained in [3,13,29,28]. A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [33,16] for a recent application to capacity o... |

1 |
Random Fibonacci sequences and the number
- Viswanath
(Show Context)
Citation Context ...ducts ∏s i=t (I − αMi) play an important role in the behavior of (1). Properties of products of random matrices are of interest in a wide range of fields. Application areas include numerical analysis =-=[15,34]-=-, statistical physics [9,10], recursive algorithms [11,5,27,17], perturbation theory for dynamical systems [1], queueing theory [23], and even botany [30]. Seminal results are contained in [3,13,29,28... |

1 |
Products of random matrices or ”why do biennials live longer than two years
- Roerdink
- 1989
(Show Context)
Citation Context ... Application areas include numerical analysis [12, 30], statistical physics [8, 9], recursive algorithms [10, 23], perturbation theory for dynamical systems [1], queueing theory [19], and even botany =-=[26]-=-. Seminal results are contained in [3, 25, 24]. 2A complementary and popular research area concerns the eigenstructure of large random matrices (see e.g. [29, 13] for recent application to capacity o... |