## Computing Minimum and Maximum Reachability Times in Probabilistic Systems (1999)

### Cached

### Download Links

- [www-cad.eecs.berkeley.edu]
- [www.soe.ucsc.edu]
- [www.soe.ucsc.edu]
- [www.cse.ucsc.edu]
- [users.soe.ucsc.edu]
- DBLP

### Other Repositories/Bibliography

Citations: | 32 - 2 self |

### BibTeX

@INPROCEEDINGS{Alfaro99computingminimum,

author = {Luca De Alfaro},

title = {Computing Minimum and Maximum Reachability Times in Probabilistic Systems},

booktitle = {},

year = {1999},

pages = {66--81},

publisher = {Springer}

}

### OpenURL

### Abstract

A Markov decision process is a generalization of a Markov chain in which both probabilistic and nondeterministic choice coexist. Given a Markov decision process with costs associated with the transitions and a set of target states, the stochastic shortest path problem consists in computing the minimum expected cost of a control strategy that guarantees to reach the target. In this paper, we consider the classes of stochastic shortest path problems in which the costs are all non-negative, or all non-positive. Previously, these two classes of problems could be solved only under the assumption that the policies that minimize or maximize the expected cost also lead to the target with probability 1. This assumption does not necessarily hold for Markov decision processes that arise as model for distributed probabilistic systems. We present efficient methods for solving these two classes of problems without relying on additional assumptions. The methods are based on algorithms to transform th...

### Citations

2604 |
Dynamic Programming
- Bellman
- 1957
(Show Context)
Citation Context ... minimum (or maximum) expected time to reach the set. 1 Introduction Markov decision processes are generalizations of Markov chains in which probabilistic choice coexists with nondeterministic choice =-=[Bel57]-=-. Several models of distributed probabilistic systems are based either on Markov decision processes [BdA95, KB98] or on closely related formalisms, such as the concurrent Markov chains of [Var85], the... |

317 | Dynamic Programming and Optimal Control - Bertsekas |

265 | Probabilistic simulations for probabilistic processes - Segala, Lynch - 1995 |

230 |
Modeling and verification of randomized distributed real-time systems
- Segala
- 1995
(Show Context)
Citation Context ...ecision processes [BdA95, KB98] or on closely related formalisms, such as the concurrent Markov chains of [Var85], the probabilistic automata of [SL94, WSS94], and the timed probabilistic automata of =-=[Seg95]-=-. Several models based on process algebras are also closely related to Markov decision processes An abbreviated version of this paper is to appear in Proceedings of CONCUR 99: Concurrency Theory. This... |

198 | Model checking of probabilistic and nondeterministic systems - Bianco, Alfaro - 1995 |

170 |
Probabilistic Automata
- Rabin
- 1963
(Show Context)
Citation Context ...on process (MDP) is a generalization of a Markov chain in which nondeterministic choice coexists with probabilistic one. Markov decision processes are closely related to the probabilistic automata of =-=[Rab63]-=-, to the concurrent Markov chains of [Var85], and to the simple probabilistic automata of [SL94, Seg95]. To present their definition, given a countable set C we denote by D(C) the set of probability d... |

158 |
M.: The complexity of probabilistic verification
- Courcoubetis, Yannakakis
- 1995
(Show Context)
Citation Context ...represent the 5 set of states and actions that can be repeated infinitely often along a path with non-zero probability. Related sets of states have been used for solving optimization problems on MDPs =-=[CY95]-=-. Given an MDP M = (S; Acts ; A; p), a sub-MDP is a pair (C; D), where C ` S is a subset of states and D : S 7! Acts is a function that associates to each s 2 S a subset D(s) ` A(s) of actions. A sub-... |

147 | Time and probability in formal design of distributed systems - Hansson - 1994 |

132 |
Finite State Markovian Decision Processes
- Derman
- 1970
(Show Context)
Citation Context ...c; once chosen, the action determines the transition probability distribution for the successor state. In order to quantify the probabilistic properties of an MDP, the concept of policy is introduced =-=[Der70]-=-, related to the schedulers of [Var85, PZ86] and to the adversaries of [SL94, Seg95]. A policy is a criterion for selecting the actions during a behavior of the system; once the policy is fixed, the M... |

115 |
Formal Verification of Probabilistic Systems
- Alfaro
- 1998
(Show Context)
Citation Context ...In this paper we present a more efficient algorithm, that solves the problem in time quadratic in the size of the MDP, and that does not require numerical computation. The algorithm, originating from =-=[dA97]-=-, is related to an algorithm for solving two-person reachability games presented in [dAHK98]. Once we have determined the states from which the target set cannot be reached with probability 1, we pres... |

115 | Model checking for a probabilistic branching time logic with fairness - Baier, Kwiatkowska - 1998 |

96 | Specification and refinement of probabilistic processes - Jonsson, Larsen - 1991 |

72 | J.N.: An analysis of stochastic shortest path problems
- Bertsekas, Tsitsiklis
- 1991
(Show Context)
Citation Context ...e Bellman operator does not necessarily admit a unique fixpoint in non-negative SSP problems not only prevents a direct application of value iteration methods, but also blocks the line of analysis of =-=[BT91]-=- for the solution based on linear programming. We present two approaches to the solution of non-negative SSP problems. The first approach is based on the observation that the difficulties in solving n... |

56 |
Markov decision processes and regular events
- Courcoubetis, Yannakakis
- 1990
(Show Context)
Citation Context ...aximum and minimum probability with which a linear-time temporal logic formula holds over an MDP [CY90, CY95, BdA95]. While the maximum reachability probability can be computed with the algorithms of =-=[CY90]-=-, the proposed approach minimizes the size of the numerical problem to be solved. 2 Preliminaries A Markov decision process (MDP) is a generalization of a Markov chain in which nondeterministic choice... |

43 | Concurrent reachability games
- Alfaro, Henzinger, et al.
(Show Context)
Citation Context ...atic in the size of the MDP, and that does not require numerical computation. The algorithm, originating from [dA97], is related to an algorithm for solving two-person reachability games presented in =-=[dAHK98]-=-. Once we have determined the states from which the target set cannot be reached with probability 1, we present two methods for solving the SSP problem on the remaining states. First, we show that non... |

34 | Symbolic model checking of concurrent probabilistic processes using MTBDDs and the Kronecker representation - Alfaro, Kwiatkowska, et al. - 2000 |

32 | Stochastic Transition Systems - Alfaro |

16 | Probabilistic verification by tableaux - Pnueli, Zuck - 1986 |

13 | Bisimulation through probabilistic testing (preliminary report - Larsen, Skou - 1989 |

11 | Optimal pursuit strategies in discrete-state probabilistic systems - Zadeh, Eaton - 1962 |