We are concerned with the allocation of the base station transmitter time in time varying mobile communications with many users who are transmitting data. Time is divided into small scheduling intervals, and the channel rates for the various users are available at the start of the intervals. Since the rates vary randomly, in selecting the current user there is a conflict between full use (by selecting the user with the highest current rate) and fairness (which entails consideration for users with poor throughput to date). The Proportional Fair Scheduler (PFS) of the Qualcomm High Data Rate (HDR) system and related algorithms are designed to deal with such conflicts. The aim here is to put such algorithms on a sure mathematical footing and analyze their behavior. The available analysis [6], while obtaining interesting information, does not address the actual convergence for arbitrarily many users under general conditions.

We present an "opportunistic" transmission scheduling policy that exploits time-varying channel conditions and maximizes the system performance stochastically under a certain resource allocation fairness constraint. We establish the optimality of the scheduling scheme and also describe a practical scheduling procedure to implement our scheme. Through simulation results, we show that the scheme also works well for nonstationary scenarios and results in performance improvements of 20--150% compared with a scheduling scheme that does not take into account channel conditions. Furthermore, we note that in wireless networks, an important role of resource allocation is to balance the system performance and fairness among "good" and "bad" users. We propose three heuristic timefraction assignment schemes, which approach the problem from different viewpoints. Keywords---Scheduling, fairness, wireless, high-rate-data. I.

It is shown here that stability of the stochastic approximation algorithm is implied by the asymptotic stability of the origin for an associated ODE. This in turn implies convergence of the algorithm. Several specific classes of algorithms are considered as applications. It is found that the results provide (i) a simpler derivation of known results for reinforcement learning algorithms; (ii) a proof for the first time that a class of asynchronous stochastic approximation algorithms are convergent without using any a priori assumption of stability; (iii) a proof for the first time that asynchronous adaptive critic and Q-learning algorithms are convergent for the average cost optimal control problem.

The Karhunen-Loeve (KL) Transform is an optimal method for approximating a set of vectors, which was used in image processing and computer vision for several tasks. Its computational demands and its batch calculation nature have limited its application. Here we present a new, sequential algorithm for calculating the KL basis, which is faster in typical applications and is especially advantageous for image sequences: the KL basis calculation is done with much lower delay and allows for dynamic updating of object databases for recognition. Systematic tests of the implemented algorithm show that these advantages are indeed obtained with the same accuracy available from batch KL algorithms. 1 Introduction The Karhunen-Loeve (KL) transform [1] is a preferred method for approximating a set of vectors by a low dimensional subspace. The method provides the optimal subspace, spanned by the KL basis, which minimizes the MSE between the given set of vectors and their projections on the subspace...

In this paper we study the ergodicity properties of some adaptive Monte Carlo Markov chain algorithms (MCMC) that have been recently proposed in the literature. We prove that under a set of verifiable conditions, ergodic averages calculated from the output of a so-called adaptive MCMC sampler converge to the required value and can even, under more stringent assumptions, satisfy a central limit theorem. We prove that the conditions required are satisfied for the Independent Metropolis-Hastings algorithm and the Random Walk Metropolis algorithm with symmetric increments. Finally we propose an application of these results to the case where the proposal distribution of the Metropolis-Hastings update is a mixture of distributions from a curved exponential family.

Congestion control algorithms used in the Internet are difficult to analyze or simulate on a large scale, i.e., when there are large numbers of nodes, links and sources in a network. The reasons for this include the complexity of the actual implementation of the algorithm and the randomness introduced in the packet arrival and service processes due to many factors such as arrivals and departures of sources and uncontrollable short flows in the network. To make the simulation tractable, often deterministic fluid model approximations of these algorithms are used. These approximations are in the form of either deterministic delay differential equations, or more generally, deterministic functional differential equations. We justify the use of deterministic models for proportionally-fair congestion controllers under a limiting regime where the number of sources in a network is large. We verify our results through simulations of window-based implementations of proportionally fair controllers and TCP.

Consider a network with an arbitrary topology and arbitrary communication delays, in which congestion control is based on additiveincrease and multiplicative-decrease. We show that the source rates tend to be distributed in order to maximize an objective function called F h A ("F h A fairness"). We derive this result under the assumption of rate proportional negative feedback and for the regime of rare negative feedback. This applies to TCP in moderately loaded networks, and to those TCP implementations that are designed to interpret multiple packet losses within one RTT as a single congestion indication and do not rely on re-transmission timeout. This result provides some insight into the distribution of rates, and hence of packet loss ratios, which can be expected in a given network with a number of competing TCP or TCP-friendly sources. We validate our findings by analyzing a multiple-bottleneck scenario, and comparing with previous results [1], [2], and an extensive numerical s...

In this paper we address the problem of the stability of the stochastic approximation procedure. The stability of such algorithms is known to rely heavily on the growth of the mean field at the boundary of the parameter set and the magnitude of the sizesteps used in the procedure. The conditions typically required to ensure convergence are either too difficult to check in practice or not satisfied at all, even for simple models. The most popular technique to circumvent this problem consists of constraining the parameter to a compact subset in the parameter space. We propose and analyze here an alternative, based on projection on adaptive truncation sets, extending previous works in this direction. This procedure allows for the adaptive tuning of the magnitude of the sizesteps, which is key to ensuring stability. The stability - with probability one - of the scheme is proved under a set of verifiable assumptions. We illustrate these claims in the so-called controlled Markovian setting and present two substantial examples. The first example is related to the minimum prediction error estimation of the parameters of stable and invertible ARMA processes and the second example is related to controlled Markov chain Monte Carlo algorithms.

Reinforcement learning is the problem of generating optimal behavior in a sequential decision-ma.king environment given the opportunity of interacting,vith it. Many algorithms for solving reinforcement-learning problems work by computing improved estimates of the optimal value function. \Ve extend prior analyses of reinforcement-learning algorithms and present a powerful new theorem that can provide a unified analysis of value-function-based reinforcement-learning algorithms. The usefulness of the theorem lies in how it allows the convergence of a complex asynchronous reinforcement-learning algorithm to be proven by verifying that a Himplcr HynchronouH algorithm convergeH. \-Ve illuHtrate the application of the theorem by analyzing the convergence of Q-learningl model-based reinforcement learning, Q-learning with multi-state updates, Q-learning for:\farkov games, and risk-sensitive reinforcement learning. 1

Congestion control algorithms used in the Internet are difficult to analyze or simulate on a large scale, i.e., when there are large numbers of nodes, links and sources in a network. The reasons for this include the complexity of the actual implementation of the algorithm and the randomness introduced in the packet arrival and service processes due to many factors such as arrivals and departures of sources and uncontrollable short flows in the network. To make the analysis or simulation tractable, often deterministic fluid approximations of these algorithms are used. These approximations are in the form of either deterministic delay differential equations, or more generally, deterministic functional differential equations (FDEs). In this paper, we ignore the complexity introduced by the window-based implementation of such algorithms and focus on the randomness in the network. We justify the use of deterministic models for proportionally-fair congestion controllers under a limiting regime where the number of flows in a network is large.