For many applications it is useful to sample from a finite set of objects in accordance with some particular distribution. One approach is to run an ergodic (i.e., irreducible aperiodic) Markov chain whose stationary distribution is the desired distribution on this set; after the Markov chain has run for M steps, with M sufficiently large, the distribution governing the state of the chain approximates the desired distribution. Unfortunately it can be difficult to determine how large M needs to be. We describe a simple variant of this method that determines on its own when to stop, and that outputs samples in exact accordance with the desired distribution. The method uses couplings, which have also played a role in other sampling schemes; however, rather than running the coupled chains from the present into the future, one runs from a distant point in the past up until the present, where the distance into the past that one needs to go is determined during the running of the al...

We present new algorithms for reinforcement learning, and prove that they have polynomial bounds on the resources required to achieve near-optimal return in general Markov decision processes. After observing that the number of actions required to approach the optimal return is lower bounded by the mixing time T of the optimal policy (in the undiscounted case) or by the horizon time T (in the discounted case), we then give algorithms requiring a number of actions and total computation time that are only polynomial in T and the number of states, for both the undiscounted and discounted cases. An interesting aspect of our algorithms is their explicit handling of the Exploration-Exploitation trade-off. 1

We construct Markov chain algorithms for sampling from discrete exponential families conditional on a sufficient statistic. Examples include generating tables with fixed row and column sums and higher dimensional analogs. The algorithms involve finding bases for associated polynomial ideals and so an excursion into computational algebraic geometry.

We quantify the effectiveness of random walks for searching and construction of unstructured peer-to-peer (P2P) networks. For searching, we argue that random walks achieve improvement over flooding in the case of clustered overlay topologies and in the case of re-issuing the same request several times. For construction, we argue that an expander can be maintained dynamically with constant operations per addition. The key technical ingredient of our approach is a deep result of stochastic processes indicating that samples taken from consecutive steps of a random walk can achieve statistical properties similar to independent sampling (if the second eigenvalue of the transition matrix is bounded away from 1, which translates to good expansion of the network; such connectivity is desired, and believed to hold, in every reasonable network and network model). This property has been previously used in complexity theory for construction of pseudorandom number generators. We reveal another facet of this theory and translate savings in random bits to savings in processing overhead.

We initiate the study of the generalization of random walks on nite graphs to the quantum world. Such quantum walks do not converge to any stationary distribution, as they are unitary and reversible. However, by suitably relaxing the denition, we can obtain a measure of how fast the quantum walk spreads or how conned the quantum walk stays in a small neighborhood. We give denitions of mixing time, lling time, dispersion time. We show that in all these measures, the quantum walk on the cycle is almost quadratically faster then its classical correspondent. On the other hand, we give a lower bound on the possible speed up by quantum walks for general graphs, showing that quantum walks can be at most polynomially faster than their classical counterparts. 1

Abstract not found

For a large class of examples arising in statistical physics known as attractive spin systems (e.g., the Ising model), one seeks to sample from a probability distribution # on an enormously large state space, but elementary sampling is ruled out by the infeasibility of calculating an appropriate normalizing constant. The same difficulty arises in computer science problems where one seeks to sample randomly from a large finite distributive lattice whose precise size cannot be ascertained in any reasonable amount of time. The Markov chain Monte Carlo (MCMC) approximate sampling approach to such a problem is to construct and run "for a long time" a Markov chain with long-run distribution #. But determining how long is long enough to get a good approximation can be both analytically and empirically difficult. Recently, Jim Propp and David Wilson have devised an ingenious and efficient algorithm to use the same Markov chains to produce perfect (i.e., exact) samples from #. However, the running t...

This article aims to provide an introductory survey on quantum random walks. Starting from a physical effect to illustrate the main ideas we will introduce quantum random walks, review some of their properties and outline their striking differences to classical walks. We will touch upon both physical effects and computer science applications, introducing some of the main concepts and language of present day quantum information science in this context. We will mention recent developments in this new area and outline some open questions. 1. Overview Ever since the discovery of quantum mechanics people have been puzzled by the counter-intuitive character of the laws of nature. Over time we have learned to accept more and more effects that are unimaginable in a classical Newtonian world. Modern technology exploits quantum effects both to our benefit and detriment—among the memorable examples we should cite laser technology and not omit the atomic bomb. In recent years interest in quantum information theory has been generated by the prospect of employing its laws to design devices of surprising power [1]. New ideas include quantum cryptography [2, 3] and quantum computation. In 1994 Shor [4] discovered a quantum algorithm to factor numbers efficiently (that is in time that grows only polynomically with the length of the number to be factored). This has unleashed a wave of activity across a broad range of disciplines: physics, computer science, mathematics and engineering. This fruitful axis of research has uncovered many new effects that are strikingly different from their classical counterparts, both from the physical point of view as well as from a computer science and communication theory perspective. Over time these communities have gained a greater understanding of the concepts and notions of the other. The idea that information cannot be separated from

In the last decade important relations between Laplace eigenvalues and eigenvectors of graphs and several other graph parameters were discovered. In these notes we present some of these results and discuss their consequences. Attention is given to the partition and the isoperimetric properties of graphs, the max-cut problem and its relation to semidefinite programming, rapid mixing of Markov chains, and to extensions of the results to infinite graphs.

CONTENTS 0 PREFACE 3 1 BASICS OF PROBABILITY THEORY 5 2 MARKOV CHAINS 12 3 COMPUTER SIMULATION OF MARKOV CHAINS 19 4 IRREDUCIBLE AND APERIODIC MARKOV CHAINS 25 5 STATIONARY DISTRIBUTIONS 30 6 REVERSIBLE MARKOV CHAINS 41 7 MARKOV CHAIN MONTE CARLO 46 8 FAST CONVERGENCE OF MCMC ALGORITHMS 54 9 APPROXIMATE COUNTING 63 10 THE PROPP--WILSON ALGORITHM 74 11 PROPP--WILSON WITH READ-ONCE RANDOMNESS 82 12 SIMULATED ANNEALING 88 13 FURTHER READING 96 1 2 0 PREFACE The first version of these lecture notes was composed for a new course on randomized algorithms at Chalmers University of Technology, in the spring semester 2000. The amount of material contained therein was not quite enough for an entire course, and the idea was to use these notes together with the book "Randomized Algorithms" by Motwani & Raghavan [MR]. The current version of 2001 has now, with the addition of three more chapters (Chapters 8, 9 and 11), expanded into something more "course-sized", bu