Abstract. Iterated random functions are used to draw pictures or simulate large Ising models, among other applications. They offer a method for studying the steady state distribution of a Markov chain, and give useful bounds on rates of convergence in a variety of examples. The present paper surveys the field and presents some new examples. There is a simple unifying idea: the iterates of random Lipschitz functions converge if the functions are contracting on the average. 1. Introduction. The

In the past twenty years there has been much interest in the physical and biological sciences in nonlinear dynamical systems that appear to have random, unpredictable behavior. One important parameter of a dynamic system is the dominant Lyapunov exponent (LE). When the behavior of the system is compared for two similar initial conditions, this exponent is related to the rate at which the subsequent trajectories diverge. A bounded system with a positive LE is one operational definition of chaotic behavior. Most methods for determining the LE have assumed thousands of observations generated from carefully controlled physical experiments. Less attention has been given to estimating the LE for biological and economic systems that are subjected to random perturbations and observed over a limited amount of time. Using nonparametric regression techniques (Neural Networks and Thin Plate Splines) it is possible to consistently estimate the LE. The properties of these methods have been studied using simulated data and are applied to a biological time series: marten fur returns for the Hudson Bay Company (1820-1900). Based on a nonparametric analysis there is little evidence for lowdimensional chaos in these data. Although these methods appear to work well for systems perturbed by small amounts of noise, finding chaos in a system with a significant stochastic component may be difficult.

By developing and applying a broad framework for rejection sampling using auxiliary randomness, we provide an extension of the perfect sampling algorithm of Fill (1998) to general chains on quite general state spaces, and describe how use of bounding processes can ease computational burden. Along the way, we unearth a simple connection between the Coupling From The Past (CFTP) algorithm originated by Propp and Wilson (1996) and our extension of Fill’s algorithm. Key words and phrases. Fill’s algorithm, Markov chain Monte Carlo, perfect sampling, exact sampling, rejection sampling, interruptibility, coupling from the past, read-once coupling from the past, monotone transition rule, realizable monotonicity, stochastic monotonicity, partially ordered set, coalescence, imputation,

This paper explores how Bayes-rational individuals learn sequentially from the discrete actions of others. Unlike earlier informational herding papers, we admit heterogeneous preferences. Not only may type-specific `herds' eventually arise, but a new robust possibility emerges: confounded learning. Beliefs may converge to a limit point where history oers no decisive lessons for anyone, and each type's actions forever nontrivially split between two actions. To verify that our identied limit outcomes do arise, we exploit the Markov-martingale character of beliefs. Learning dynamics are stochastically stable near a fixed point in many Bayesian learning models like this one.

Algorithms for perfect or exact simulation of random samples from the invariant measure of a Markov chain have received considerable recent attention following the introduction of the "coupling-from-the-past" (CFTP) technique of Propp and Wilson. Here we place such algorithms in the context of backward coupling of stochastically recursive sequences. We show that although general backward couplings can be constructed for chains with finite mean forward coupling times, and can even be thought of as extending the classical "Loynes schemes" from queueing theory, successful "vertical" CFTP algorithms such as those of Propp and Wilson can be constructed if and only if the chain is uniformly geometric ergodic. We also relate the convergence moments for backward coupling methods to those of forward coupling times: the former typically lose at most one moment compared to the latter. Work supported in part by NSF Grant DMS-9504561 and by CRDF Grant RM1-226 y Postal Address: Institute of Math...

We develop an algorithm for simulating approximate random samples from the invariant measure of a Markov chain using backward coupling of embedded regeneration times. Related methods have been used effectively for finite chains and for stochastically monotone chains: here we propose a method of implementation which avoids these restrictions by using a "cycle-length" truncation. We show that the coupling times have good theoretical properties and describe benefits and difficulties of implementing the methods in practice. 1 Introduction There has been considerable recent work on the development and application of algorithms that will enable the simulation of the invariant measure ß of a Markov chain, either exactly (that is, by drawing a random sample known to be from ß) or approximately, but with computable order of accuracy. These were sparked by the seminal paper of Propp and Wilson [18], and several variations and extensions of this idea have appeared in the literature including rece...

this paper, has been directed toward extending their results beyond Axiom A.

We discuss synchronous and asynchronous iterations of the form x k+1 = x k + γ(k)(h(x k)+w k), where h is a suitable map and {wk} is a deterministic or stochastic sequence satisfying suitable conditions. In particular, in the stochastic case, these are stochastic approximation iterations that can be analyzed using the ODE approach based either on Kushner and Clark’s lemma for the synchronous case or on Borkar’s theorem for the asynchronous case. However, the analysis requires that the iterates {xk} be bounded, a factwhich is usually hard to prove. We develop a novel framework for proving boundedness in the deterministic framework, which is also applicable to the stochastic case when the deterministic hypotheses can be verified in the almost sure sense. This is based on scaling ideas and on the properties of Lyapunov functions. We then combine the boundedness property with Borkar’s stability analysis of ODEs involving nonexpansive mappings to prove convergence (with probability 1 in the stochastic case). We also apply our convergence analysis to Q-learning algorithms for stochastic shortest path problems and are able to relax some of the assumptions of the currently available results.

We give an overview of the ideas central to some recent developments in the ergodic theory of the stochastically forced Navier Stokes equations and other dissipative stochastic partial differential equations. Since our desire is to make the core ideas clear, we will mostly work with a specific example: the stochastically forced Navier Stokes equations. To further clarify ideas, we will also examine in detail a toy problem. A few general theorems are given. Spatial regularity, ergodicity, exponential mixing, coupling for a SPDE, and hypoellipticity are all discussed. This article attempts to collect a number of ideas which have proven useful in the study of stochastically forced dissipative partial differential equations. The discussion will center around those of ergodicity but will also touch on the regularity of both solutions and transition densities. Since our desire is to make the core ideas clear, we will mostly work with a specific example: the stochastically forced Navier Stokes equations. To further clarify ideas, we will also examine in detail a toy problem. Though we have not tried to give any great generality, we also present a number of abstract results to help isolate what assumptions are used in which arguments. Though a few results are presented in new ways and a number of proofs are streamlined, the core ideas remain more or less the same as in the originally cited papers. We do improve sightly the exponential mixing results given in [Mat02c]; however, the techniques used are the same. Lastly, we do not claim to be exhaustive. This is not meant to be an all encompassing review article. The view point given here is a personal one; nonetheless, citations are given to good starting points for related works both by the author and others. Consider the two-dimensional Navier-Stokes equation with stochastic forcing:

. We consider random piecewise smooth, piecewise invertible maps mainly on the interval but also in higher dimensions. We assume that, on the average and possibly without any stochastic uniformity: (i) the maps expand distances, (ii) have not too many pieces, (iii) have not too large a distortion, and (iv) are strongly mixing. We assume no Markov property. We prove that as in the classical case of the iteration of a fixed piecewise expanding map of the interval, we have exponential decay of random correlations. Our proof builds on the one given by C. Liverani for deterministic, mixing and piecewise expanding interval maps. We demand very little of the stochastic process giving the maps. In particular, if the maps are fi-transformations on [0; 1[ d , i.e., xn+1 = Bn+1xn mod Z d with Bn : R d ! R d affine, then our results apply to all stationary and ergodic processes B 1 ; B 2 ; : : : which expand on the average and satisfy the mixing condition above. We remark that our settin...