## Recursive Markov decision processes and recursive stochastic games (2005)

### Cached

### Download Links

- [homepages.inf.ed.ac.uk]
- [homepages.inf.ed.ac.uk]
- DBLP

### Other Repositories/Bibliography

Venue: | In Proc. of 32nd Int. Coll. on Automata, Languages, and Programming (ICALP’05 |

Citations: | 35 - 9 self |

### BibTeX

@INPROCEEDINGS{Etessami05recursivemarkov,

author = {Kousha Etessami and Mihalis Yannakakis},

title = {Recursive Markov decision processes and recursive stochastic games},

booktitle = {In Proc. of 32nd Int. Coll. on Automata, Languages, and Programming (ICALP’05},

year = {2005},

pages = {891--903},

publisher = {Springer}

}

### OpenURL

### Abstract

Abstract. We introduce Recursive Markov Decision Processes (RMDPs) and Recursive Simple Stochastic Games (RSSGs), and study the decidability and complexity of algorithms for their analysis and verification. These models extend Recursive Markov Chains (RMCs), introduced in [EY05a,EY05b] as a natural model for verification of probabilistic procedural programs and related systems involving both recursion and probabilistic behavior. RMCs define a class of denumerable Markov chains with a rich theory generalizing that of stochastic context-free grammars and multi-type branching processes, and they are also intimately related to probabilistic pushdown systems. RMDPs & RSSGs extend RMCs with one controller or two adversarial players, respectively. Such extensions are useful for modeling nondeterministic and concurrent behavior, as well as modeling a system’s interactions with an environment. We provide a number of upper and lower bounds for deciding, given an RMDP (or RSSG) A and probability p, whether player 1 has a strategy to force termination at a desired exit with probability at least p. We also address “qualitative ” termination questions, where p = 1, and model checking questions. 1

### Citations

581 | Markov Decision Processes - Puterman - 1994 |

556 | Matrix Geometric Solutions in Stochastic Models. An Algorithmic Approach - Neuts - 1981 |

453 | Branching processes - Athreya, Ney - 1972 |

369 |
The Theory of Branching Processes
- Harris
- 1963
(Show Context)
Citation Context ...rsive manner. RMCs define a class of denumerable Markov chains with a rich theory generalizing that of Stochastic Context-Free Grammars (SCFGs) (see, e.g., [MS99]) and Multi-Type Branching Processes (=-=[Har63]-=-), both of which correspond to 1-exit RMCs: RMCs in which each component Markov chain has 1 terminating exit stateswhere it can return control back to a component that called it. RMCs are also intimat... |

222 |
Competitive Markov Decision Processes
- Filar, Vrieze
- 1997
(Show Context)
Citation Context ...ively in recent years for verification of probabilistic systems. Stochastic games generalize MDPs with multiple players, and in their 2-player zero-sum version are also known as Competitive MDPs (see =-=[FV97]-=-). Simple Stochastic Games (SSGs) [Con92] are a special class of 2-player zero-sum stochastic games, where the goal of one player is to reach a given terminal state, while the other aims to avoid it. ... |

215 | Automatic verification of probabilistic concurrent finite state programs - Vardi - 1985 |

213 | Tree automata, mu-calculus and determinacy - Emerson, Jutla |

196 | On the combinatorial and algebraic complexity of quantifier elimination - Basu, Pollack, et al. - 1996 |

173 | On the computational complexity and geometry of the first-order theory of the reals - Renegar - 1992 |

160 |
Introduction to probabilistic automata
- Paz
- 1971
(Show Context)
Citation Context ...roofs: one is a direct reduction from the halting problem for 2-counter machines (see, e.g., [HU79]), and the second is a reduction from the emptiness problem for Probabilistic Finite Automata (PFAs) =-=[Paz71]-=-. The latter reduction is simpler and has the advantage that it connects the well-studied area of PFAs to RMDPs (we show that PFAs can be seen, in effect, as a special case of RMDPs) and also allows u... |

158 | M.: The complexity of probabilistic verification - Courcoubetis, Yannakakis - 1995 |

151 | The complexity of stochastic games
- Condon
- 1992
(Show Context)
Citation Context ...f probabilistic systems. Stochastic games generalize MDPs with multiple players, and in their 2-player zero-sum version are also known as Competitive MDPs (see [FV97]). Simple Stochastic Games (SSGs) =-=[Con92]-=- are a special class of 2-player zero-sum stochastic games, where the goal of one player is to reach a given terminal state, while the other aims to avoid it. SSGs generalize parity games and other im... |

134 | Pushdown processes: games and model checking - Walukiewicz |

126 | Some algebraic and geometric computations in pspace - Canny - 1988 |

123 | Stochastic context-free grammars for tRNA modeling - Sakakibarra, Brown, et al. - 1994 |

122 | V.: Introduction to matrix analytic methods in stochastic modeling - Latouche, Ramaswami - 1999 |

117 | Matrix Analysis,” Cambridge - Horn, Johnson - 1985 |

115 | Model checking for a probabilistic branching time logic with fairness - Baier, Kwiatkowska - 1998 |

110 | Analysis of recursive state machines
- ALUR, BENEDIKT, et al.
- 2005
(Show Context)
Citation Context ...where V2 = ∅ is called a Recursive Markov Decision Process (RMDP); an RSSG where V1 ∪ V2 = ∅ is called a Recursive Markov Chain ([EY05a,EY05b]); an RSSGswhere V0 ∪ V2 = ∅ is called a Recursive Graph (=-=[AEY01]-=-); an RSSG where V0 = ∅ is called a Recursive Game Graph (see [ATM03,Ete04]). We use 1-exit RSSG to refer to RSSGs where every component has 1 exit. Without loss of generality, we can assume that ever... |

96 |
Branching processes with biological applications
- Jagers
(Show Context)
Citation Context ...y in the work of Kolmogorov, Sevastianov, Harris and others for MT-BPs and beyond (see, e.g., [Har63]). These have been used to model a wide variety of applications, including in population genetics (=-=[Jag75]-=-), nuclear chain reactions ([EU48]), and RNA modeling in computational biology (based on SCFGs) ([SBH + 94]). SCFGs are also fundamental models in statistical natural language processing (see, e.g., [... |

76 | Positional strategies for mean payoff games - Ehrenfeucht, Mycielski - 1979 |

75 | Adversarial queuing theory - Borodin, Kleinberg, et al. |

68 | Recursive markov chains, stochastic grammars, and monotone systems of nonlinear equations
- Etessami, Yannakakis
(Show Context)
Citation Context ... equations generalize linear Bellman’s equations for finite MDPs (see, e.g., [Put94,FV97]) and they also generalize the monotone systems of nonlinear polynomial equations for RMCs that we studied in (=-=[EY05a]-=-). – We show a quite nontrivial Stackless & Memoryless (S&M) Determinacy result for 1-exit RSSG termination, whereas we observe this fails badly even for 2-exit RMDPs (namely, we observe that optimal ... |

65 |
The determinacy of Blackwell games
- Martin
- 1998
(Show Context)
Citation Context ... given instead a strategy τ ∈ Ψ2, let q k,·,τ (u,ex) = maxσ∈Ψ1 q k,σ,τ (u,ex) , and let q∗,·,τ (u,ex) = supσ∈Ψ1 q∗,σ,τ (u,ex) . From very general determinacy results (Martin’s “Blackwell determinacy” =-=[Mar98]-=- is one such result which applies to all two-player zero-sum stochastic games with countable state spaces) it follows that the games MA are determined, meaning that supσ∈Ψ1 infτ ∈Ψ2 q ∗,σ,τ (u,ex) = i... |

63 | Model checking probabilistic pushdown automata - Kucera, Esparza, et al. |

56 |
Markov decision processes and regular events
- Courcoubetis, Yannakakis
- 1990
(Show Context)
Citation Context ...udied in both finite state and infinite state variants. Verification of finite state MDPs, also called concurrent Markov chains, has been studied for a long time (see, e.g., [CY98,CY95,Var85,HSP83]). =-=[CY98]-=- provides efficient algorithms for ω-regular model checking of finite MDPs. Model checking tools like PRISM contain optimized implementations of branching-time model checkers for finite MDPs (see, e.g... |

54 | On the complexity of space bounded interactive proofs (extended abstract - Condon, Lipton - 1989 |

53 | Termination of probabilistic concurrent programs - Hart, Sharir, et al. - 1983 |

48 | Model checking for probability and time: from theory to practice
- Kwiatkowska
(Show Context)
Citation Context ...finite MDPs have been studied extensively for verification of probabilistic systems, and optimized verification tools already exist for them (see, e.g.,[CY98,CY95,Var85,HSP83,BK98,dAKN + 00], and see =-=[Kwi03]-=- for a recent survey). SSGs extend MDPs further with a second (adversarial) player. Like non-probabilistic game graphs, they can also be used to model and analyze the interactions between a controlled... |

48 | On the undecidability of probabilistic planning and related stochastic optimization problems - Madani, Hanks, et al. |

45 | On the complexity of numerical analysis - Allender, Bürgisser, et al. |

37 | Algorithmic verification of recursive probabilistic state machines - Etessami, Yannakakis - 2005 |

36 | Distinguishing tests for nondeterministic and probabilistic machines
- Courcoubetis, C, et al.
- 1995
(Show Context)
Citation Context ... the RMDP A that is constructed above from a PFA M into another RMDP A ′ . (Note that the qualitative problem for PFAs, i.e. determining whether p ∗ M = 1, is decidable; this follows from a result of =-=[ACY95]-=-.) The RMDP A ′ has one more component, and the embedding has the property that it turns the termination probability from 1 − ǫ into 1; see the appendix for details. ⊓⊔ We next show undecidability of ... |

34 | Undecidable Problems for Probabilistic Automata of Fixed Dimension
- Blondel
(Show Context)
Citation Context ...sely p∗ M = sup{PM(w)|w ∈ Σ ∗ } (see the appendix). Thus, for a threshold λ, the language L(M, λ) = ∅ iff q∗ (en,ex) ≤ λ; this establishes the undecidability of the quantitative problem for RMDPs. In =-=[BC03]-=- it is shown that the PFA emptiness problem is undecidable even for PFAs with only 2 letters and 46 states. It follows that the quantitative problem is undecidablesfor RMDPs with one component and 46 ... |

32 | An inequality for the discriminant of a polynomial - Mahler - 1964 |

30 | On the decidability of temporal properties of probabilistic pushdown automata - Brázdil, Kučera, et al. - 2005 |

25 | Symbolic model checking of probabilistic processes using mtbdds and the kronecker representation - Alfaro, Kwiatkowska, et al. - 2000 |

17 | Recursive concurrent stochastic games - Etessami, Yannakakis - 2006 |

15 | D.: Branching Processes in Biology - Kimmel, Axelrod - 2002 |

14 | The complexity of mean payoff games on graphs. Theoretical Computer Science, 158:343–359 - Zwick, Paterson - 1996 |

12 | Verifying probabilistic procedural programs
- Esparza, Etessami
- 2004
(Show Context)
Citation Context ... are intimately related to probabilistic Pushdown Systems (pPDSs). Model checking questions for pPDSs, for both linear and branching time properties, have also been recently studied in [EKM04,BKS05]. =-=[EE04]-=- is an early survey paper which summarizes the results in only the papers [EKM04,EY05a,BKS05]. RMDPs and RSSGs are natural extensions of RMCs, introducing nondeterministic and game behavior. Countable... |

12 | Efficient qualitative analysis of classes of recursive Markov decision processes and simple stochastic games - Etessami, Yannakakis - 2006 |

11 | Modular strategies for recursive game graphs
- Alur, Torre, et al.
- 2006
(Show Context)
Citation Context ...l a strategy Stackless & Memoryless (S&M) if it is not only independent of the history of the game, but also independent of the current call stack, i.e., only depends on the current vertex. (See also =-=[ATM03]-=-, where such strategies are called modular strategies.) Corollary 1. In every 1-exit RSSG termination game, player 2 (the minimizer) has an optimal S&M strategy. Proof. Consider the strategy τ ′ in th... |

11 | A.: Random walks with “back buttons” (extended abstract - Fagin, Karlin, et al. |

10 | Recursive Games. In Contributions to the Theory of Games - Everett - 1957 |

9 |
Introduction to Automata Theory, Formal Languages and Computation
- Hopcroft, Ullman
- 1979
(Show Context)
Citation Context ... either q∗ (en,ex) = 1 or q∗ (en,ex) < ǫ, it is undecidable to distinguish which is the case. We have two proofs: one is a direct reduction from the halting problem for 2-counter machines (see, e.g., =-=[HU79]-=-), and the second is a reduction from the emptiness problem for Probabilistic Finite Automata (PFAs) [Paz71]. The latter reduction is simpler and has the advantage that it connects the well-studied ar... |

9 | Quasibirth-death processes, tree-like QBDs, probabilistic 1-counter automata, and pushdown systems - Etessami, Wojtczak, et al. - 2008 |

5 | On the existence of stationary optimal strategies - Ornstein - 1969 |

4 | Analysis of recursive game graphs using data flow equations - ETESSAMI - 2004 |

3 |
Multiplicative systems, part i., ii, and iii
- Everett, Ulam
- 1948
(Show Context)
Citation Context ...ianov, Harris and others for MT-BPs and beyond (see, e.g., [Har63]). These have been used to model a wide variety of applications, including in population genetics ([Jag75]), nuclear chain reactions (=-=[EU48]-=-), and RNA modeling in computational biology (based on SCFGs) ([SBH + 94]). SCFGs are also fundamental models in statistical natural language processing (see, e.g., [MS99]). 1-exit RMDPs correspond to... |