## Value Iteration and Optimization of Multiclass Queueing Networks (1997)

Venue: | Queueing Systems |

Citations: | 29 - 9 self |

### BibTeX

@ARTICLE{Chen97valueiteration,

author = {Rong-rong Chen and Sean Meyn},

title = {Value Iteration and Optimization of Multiclass Queueing Networks},

journal = {Queueing Systems},

year = {1997},

volume = {32},

pages = {65--97}

}

### Years of Citing Articles

### OpenURL

### Abstract

. This paper considers in parallel the scheduling problem for multiclass queueing networks, and optimization of Markov decision processes. It is shown that the value iteration algorithm may perform poorly when the algorithm is not initialized properly. The most typical case where the initial value function is taken to be zero may be a particularly bad choice. In contrast, if the value iteration algorithm is initialized with a stochastic Lyapunov function, then the following hold (i): A stochastic Lyapunov function exists for each intermediate policy, and hence each policy is regular (a strong stability condition). (ii): Intermediate costs converge to the optimal cost. (iii): Any limiting policy is average cost optimal. It is argued that a natural choice for the initial value function is the value function for the associated deterministic control problem based upon a fluid model, or the approximate solution to Poisson's equation obtained from the LP of Kumar and Meyn. Numerical studi...

### Citations

581 |
Markov Decision Processes
- Puterman
- 1994
(Show Context)
Citation Context ...counterexamples have appeared since the early sixties, with most of the general positive results holding in the case of finite state space models only. A thorough survey is found on pages 429--433 of =-=[Put94]-=-. In early papers the analyses typically focus on the differential cost function g n (x) = Vn+1 (x) \Gamma Vn (x) and the normalized value function hn (x) = Vn (x) \Gamma Vn (`), where ` 2 RONG-RONG C... |

225 | On positive Harris recurrence of multiclass queueing networks: A unified approach via fluid limit models - Dai - 1995 |

110 | Stability of queueing networks and scheduling policies
- Kumar, Meyn
- 1995
(Show Context)
Citation Context ...se the latter value function to initialize the value iteration algorithm. A second approach we consider is based on computing an approximate solution to Poisson's equation through the stability LP of =-=[KM95]-=-. Results from numerical experiments show that either choice may lead to fast convergence to an optimal policy. We thus arrive at a new way of using the information gained from solving a deterministic... |

100 | Dynamic instabilities and stabilization methods in distributed real-time scheduling of manufacturing systems - Kumar, Seidman - 1990 |

78 | Stability and convergence of moments for multiclass queuing networks via fluid limit models - Dai, Meyn - 1995 |

75 | Discrete-time controlled markov processes with average cost criterion: a survey - Arapostathis, Borkar, et al. - 1993 |

62 |
Scheduling networks of queues: heavy traffic analysis of a simple open network. Queueing Systems: Theory and Applications, 5:265--280
- Harrison, Wein
- 1989
(Show Context)
Citation Context ...nt paper [MSS96] treats the optimal control of a multiclass queueing network by relating this problem to the optimal control of an associated diffusion process in heavy traffic, following the work of =-=[HW89]-=-. Methods for translating an optimal policy for the Brownian system model back to an implementable policy for the discrete-stochastic model are introduced in [Har96]. In [Mey97b, Mey96] it is shown th... |

61 | Performance bounds for queueing networks and scheduling policies - Kumar, Kumar - 1994 |

58 |
Discrete flow networks: Bottleneck analysis and fluid approximations
- Chen, Mandelbaum
- 1991
(Show Context)
Citation Context ...ons to the differential equation d dt OE(t) = K X i=0si [e i+1 \Gamma e i ]u i (t); a.e. t 2 R+ ; (3.3) where the function u(t) is analogous to the discrete control, and satisfies similar constraints =-=[CM91]-=-. The fluid limit model L is called L p -stable if lim t!1 sup OE2L E[jOE(t)j p ] = 0: It is shown in [KM96, Mey97] that L 2 -stability of the fluid limit model is equivalent to a form of c-regularity... |

48 | Duality and Linear Programs for Stability and Performance Analysis of Queueing Networks and Scheduling Policies
- Kumar
- 1996
(Show Context)
Citation Context ...r all such (ff; fi). This strong connection between the existence of a Lyapunov function, and the existence of a bound on steady state performance is precisely the principle of duality established in =-=[KM96]-=-. If vectors (ff; fi) exist which satisfy these constraints, then the simultaneous Lyapunov condition of Hordijk is also satisfied [Hor77]. Unfortunately, in many examples of interest it is not possib... |

43 | A Liapunov bound for solutions of the Poisson equation
- Glynn, Meyn
- 1996
(Show Context)
Citation Context ...'s equation may be explicitly written as h(x) = GKc (x) = 1 X i=0 (K \Gamma s\Omega ) i Kc (x); (A.3) wheresc(x) = c(x) \Gamma (c), provided the sum is absolutely convergent [GM96, Mey97b]. The paper =-=[GM96]-=- uses these ideas to establish the following sufficient condition for the existence of suitably bounded solutions to Poisson's equation. Define the set S by S = fx : Kc (x)s(c)g: (A.4) If the chain is... |

33 | Stability of multiclass queueing networks under priority service disciplines
- Chen, Zhang
- 1995
(Show Context)
Citation Context ...atic for any such policy, and the accessibility Assumption (A3) holds with ` equal to the empty state [Mey97]. There are many stabilizing policies which may serve as the initial policy w \Gamma1 (see =-=[CZ96]-=-). This leads to several algorithmic approaches to constructing the initial value function V 0 based on the value function for the fluid model. We begin with the following suggestive proposition. The ... |

29 |
The BIGSTEP approach to flow management in stochastic processing networks
- Harrison
- 1996
(Show Context)
Citation Context ...eavy traffic, following the work of [HW89]. Methods for translating an optimal policy for the Brownian system model back to an implementable policy for the discrete-stochastic model are introduced in =-=[Har96]-=-. In [Mey97b, Mey96] it is shown that the value function for the network scheduling problem can be approximated by the value function for an associated fluid limit model. Some heuristics based upon th... |

20 |
Dynamic Programming and Markov Potential Theory
- Hordijk
- 1977
(Show Context)
Citation Context ...nary policy; and the almost monotone condition on the cost function of [Bor91]. The irreducibility assumption was relaxed in [CF95] by imposing a global Lyapunov function condition similar to that of =-=[Hor77]-=-. The global Lyapunov function condition is expressed as, E w [V (\Phi(t + 1)) j \Phi(t) = x]sV (x) \Gamma c(x; w(x)) + b1l S (x); t 2 Z+ ; (1.3) where V is a positive function on the state space, b !... |

19 |
Topics in controlled Markov chains
- Borkar
- 1991
(Show Context)
Citation Context ...nvergence holds under two natural assumptions: the stabilizability condition that the steady state cost is finite for some stationary policy; and the almost monotone condition on the cost function of =-=[Bor91]-=-. The irreducibility assumption was relaxed in [CF95] by imposing a global Lyapunov function condition similar to that of [Hor77]. The global Lyapunov function condition is expressed as, E w [V (\Phi(... |

17 | Heavy traffic convergence of a controlled, multiclass queueing system - MARTINS, SHREVE, et al. - 1996 |

16 | Fluid network models: Linear programs for control and performance bounds
- Eng, Humphrey, et al.
- 1996
(Show Context)
Citation Context ...y deterministic then we may use the limiting function V (x) = bjxj 2 Z 1 0 jOE( )j d; OE(0) = x jxj : (5.2) We can also obtain an approximation to (5.1) based upon a finite dimensional linear program =-=[DEM96]-=-. If a sufficiently tight approximation to (5.1) is found then one can expect that (3.4) will hold for this approximation. We illustrate this approach with the three buffer example illustrated in Figu... |

16 |
The policy improvement algorithm for Markov decision processes with general state space
- Meyn
- 1997
(Show Context)
Citation Context ...ish conditions for convergence which are valid in the networks context. Both the assumptions imposed and the methods of analysis are based on the recent treatment of the policy iteration algorithm of =-=[Mey97b]. The-=- major contribution of this paper is to resolve a significant drawback to the value iteration approach - it can be extremely slow. On page 385 of [Put94] the author writes "In average reward mode... |

7 | Scheduling of Multiclass Queueing Networks: Bounds on Achievable Performance", Master of Science in Electrical Engineering and - Paschalidis - 1992 |

6 |
Stability and optimization of multiclass queueing networks and their fluid models
- Meyn
- 1997
(Show Context)
Citation Context ...shown that the value function for the network scheduling problem can be approximated by the value function for an associated fluid limit model. Some heuristics based upon this result are developed in =-=[Mey97]-=- to translate a policy for the fluid model back to the original discrete network. The results reported here provide a more exact approach to translating an optimal policy for the fluid model back to t... |

4 |
On the ergodicity of stohastic processes describing the operation of open queueing networks
- Rybko, Stolyar
- 1992
(Show Context)
Citation Context ...s a model introduced independently in the papers [RS92, KS90]. Consider the network illustrated in Figure 4 consisting of four buffers and two machines fed by separate arrival streams. It is shown in =-=[RS92]-=- that the last buffer-first served policy where buffers 2 and 4 receive priority at their respective machines will make the controlled process \Phi transient, even when the load conditions (3.2) are s... |

3 | Risk sensitive optimal control: Existence and synthesis for models with unbounded cost
- Borkar, Meyn
- 1998
(Show Context)
Citation Context ...erage costs J(w n ; x) converge to the optimal cost jsas n !1; and that any limiting policy is average cost optimal. Some of these ideas have been generalized to the risk sensitive control problem/in =-=[BM98]-=-. In the network optimization problem the relative value function for the optimal policy may be approximated by the value function for the associated fluid control problem. It is thus natural to use t... |

3 |
Value iteration in a class of communicating Markov decision chains with the average cost criterion. Technical report, Universidad Aut'onoma Agraria Anonio Narro
- Cavazos-Cadena
- 1996
(Show Context)
Citation Context ... a solution of the average cost optimality equation (see (2.1,2.2) below). Recently there has been a resurgence of interest in understanding the algorithm when the state space is unbounded. The paper =-=[Cav96]-=- treats countable state space models where the state space is a single communication class under any stationary policy. Convergence holds under two natural assumptions: the stabilizability condition t... |

3 |
Value iteration in a class of average controlled Markov chains with unbounded costs: Necessary and sufficient conditions for pointwise convergence
- Cavazos-Cadena, Fernandez-Gaucherand
- 1995
(Show Context)
Citation Context ...bilizability condition that the steady state cost is finite for some stationary policy; and the almost monotone condition on the cost function of [Bor91]. The irreducibility assumption was relaxed in =-=[CF95]-=- by imposing a global Lyapunov function condition similar to that of [Hor77]. The global Lyapunov function condition is expressed as, E w [V (\Phi(t + 1)) j \Phi(t) = x]sV (x) \Gamma c(x; w(x)) + b1l ... |

3 |
A new condition for the existence of optimal stationary policies in average cost Markov decision processes
- Sennott
- 1986
(Show Context)
Citation Context ...oughout the paper in our assumptions and in the analysis to follow. These conditions will be met under the stabilizability part of Assumption (A1), and Assumptions (A2), (A3) below (the conditions of =-=[Sen86]-=- may be verified, following the approach of [Mey97b]). Assumptions (A2) and (A3) are related to the near-monotone assumption of [Bor91]. We call a function c norm-like if the sublevel set fx : c(x)sbg... |

3 |
The convergence of value iteration in average cost Markov decision chains
- Sennott
- 1996
(Show Context)
Citation Context ...ngleton. Under this assumption it may be shown that E v x [ ` ] is uniformly bounded over all policies, for each initial condition x, wheres` is the first return time to the state ` 2 X. In the paper =-=[Sen96]-=-, conditions are determined under which the optimal cost jsis computable through the limit js= lim n!1 Vn (x)=n, x 2 X. The analysis is based upon the discounted control problem, and the use of a trun... |

2 |
Optimal draining of a fluid re-entrant line. In Stochastic Networks, volume 71 of IMA volumes in Mathematics and its Applications, pages 91--103
- Weiss
- 1995
(Show Context)
Citation Context ...he parameterss= 0:1429;s1 = 0:3492;s2 = 0:1587;s3 = 0:3492 (3.5) so that ae 1 = = 1 + = 3 = 9=10, and ae 2 = = 2 = 9=11. The optimal policy for the fluid model illustrated in Figure 2 was computed in =-=[Wei95]-=-. It can be defined succinctly as Serve buffer one if and only if x 3 = 0, or x 1sx 3 \Gamma 1+(1l(x 2 = 0)) \Gamma1 . The form of the policy is logical: If the second buffer is non-empty, then the la... |

1 | The policy improvement algorithm: General theory with applications to queueing networks and their fluid models - Meyn - 1996 |