Citation Context

...uitable when the transitions are so arbitrary as to make the notion of states meaningless. This model has been extensively studied in the context of repeated games [17], prediction with expert advice =-=[18]-=- and the adversarial multiarmed bandit [7]. In MDPs with arbitrarily varying transition functions, but a fixed reward function, [6] proposes a solution using robust dynamicprogramming.However,thismeth...

Citation Context

...work considers a model more general than that of [10]. We give comparisons and pose open problems in Section V. II. SETTING A Markov decision process is a standard sequential decision-making problem (=-=[11]-=-, [12]). At each time step t ∈ {1, 2, . . .}, an agent takes an action at from a finite set A. Starting from a fixed state s1, the agent occupies at each time step t a state st belonging to a finite s...

Citation Context

... in [10], the ORDP algorithm can be modified to cope with limited observation of the reward functions and to reduce computational complexity. The first modification allows a bandit-like setting as in =-=[7]-=-, where instead of observing the full reward function rt after time step t, the agent observes only the actual reward rt(st, at) that it received. The second modification provides a way to tradeoff co...

Citation Context

...DP algorithm. For the purpose of analysis, we consider the synchronous version of the Q-learning iteration in the Q-FPL algorithm and write it in the form of a stochastic approximation algorithm (cf. =-=[22]-=-): Qt+1 = Qt + γm(ht(Qt) + Mt+1), t ∈ αm, ht(Q)(s, a) = rtm(s, a) + ∑ Pt(s ′ | s, a) max a ′′ ∈A Q(s′ , a ′′ ) s ′ ∈S − Q(s, a) − Q(s0, a0), Mt+1(s, a) = max a ′ ∈A Qt(st+1, a ′ ) − ∑ s ′ ∈S Pt(s ′ |...

Citation Context

...metry in the extent of arbitrary uncertainty in the rewards and transitions, as illustrated by the following game. Example II.3 (AR-AT Games). Additive-reward additivetransition stochastic games (cf. =-=[15]-=-) can provide natural ex-1 − ǫ 1 − ǫ 1 1 − ǫ ǫ ǫ ǫ ǫ/2 ǫ/2 ǫ ǫ 1 ǫ/2 ǫ/2 ǫ 3 2 4 1 − ǫ 1 − ǫ (a) Transition model Pt(· | ·, L) at even time steps. 1 − ǫ 3 2 4 1 − ǫ 1 − ǫ (b) Transition model Pt(· | ...

Citation Context

...e.g., the next-state distribution following each arm-selection is only specified within some range. This uncertainty may arise from inaccurate measurement or estimation, from an uncertainty principle =-=[13]-=-, or from nonstationaryvariationsin the probabilities. The latter occur, for instance, in systems that rely on replaceable components whose specifications fall within some ǫ-tolerance range (e.g., bat...

Citation Context

...edby Qαm−1 andusedto derivea new policy(cf.(3))tobeusedthroughoutthenextinterval αm.As in the ORDP algorithm,this policy is computed accordingto the concept of “following the perturbed leader” ([17], =-=[21]-=-), where continuity between successive policies is ensured by adding a vanishing noise term nt to the Q-function Qαm−1 . However, the Q-FPL algorithm is computationally more efficient: at each time st...

Citation Context

...a notion of state. Such a model is suitable when the transitions are so arbitrary as to make the notion of states meaningless. This model has been extensively studied in the context of repeated games =-=[17]-=-, prediction with expert advice [18] and the adversarial multiarmed bandit [7]. In MDPs with arbitrarily varying transition functions, but a fixed reward function, [6] proposes a solution using robust...

Citation Context

...(s ′ | s, a) max a ′′ ∈A Qt(s ′ , a ′′ ), where the state st+1 is distributed according to Pt(· | s, a). Our analysis of synchronous Q-learning can be extended to the asynchronous version, as done in =-=[23]-=-. In contrast to the convergence analysis for average-reward Q-learning ([23]), the function ht is not fixed, but may change arbitrarily in our setting. We also define the sequence Qδ t with the fixed...

Citation Context

...to the system (probabilistic transitions, uncertainty principles), or may arise from measurements. In many decision-making problems, uncertainty exists in the rewards and transition probabilities (cf.=-=[6]-=- andreferencestherein).When thisuncertainty follows a stochastic model ([7], [8]), sampling can give estimatesonthe parametersofthe transitionmodel,but some residual uncertainty always remains as a re...

Citation Context

...from measurements. In many decision-making problems, uncertainty exists in the rewards and transition probabilities (cf.[6] andreferencestherein).When thisuncertainty follows a stochastic model ([7], =-=[8]-=-), sampling can give estimatesonthe parametersofthe transitionmodel,but some residual uncertainty always remains as a result of limited samples. Under these circumstances, if it is imperative to take ...

Citation Context

...h state space S,action space A,an uncertainty set D ⊆ ∆(C), and with reward function brt. In other words, solve the following Bellman equations for MDPs with infinite-horizon average-reward objective =-=[19]-=- (via linear programming or otherwise): Vt(s) = max a∈A “ brt(s, a) + inf δ∈D X Vt(s ′ ) X δ(c)P c (s ′ ” | st, a) − Vt(s0) , s ∈ S, (2) s ′ ∈S where s0 ∈ S is a fixed state and Vt(s0) is a normalizat...

Citation Context

...ture). This generalized version is reminiscent of stochastic games [1]. However, in stochastic games, one usually assumes that there is an adversary whose utility is well-defined. In our setup, as in =-=[2]-=-, [3], and more generally in the online learning setting [4], we do not assume that such an adversary exists, but rather that the reward and transition processes are modulated 1 by arbitrary individua...

Citation Context

...otions have previously been studied separately. Online learning yields solutions that are robust against arbitrary variations in the reward functions when the transition probabilities are fixed ([2], =-=[9]-=-). Robust dynamic programming has been used to control MDPs where the transition probabilities may vary arbitrarily, but where the reward functions may not [6]. In this work, we address both uncertain...

Citation Context

...se specifications fall within some ǫ-tolerance range (e.g., battery life, physical dimensions). Our model also encompasses MDPs where the transition functionmay change abruptlyat unknowntime instants =-=[14]-=-, e.g., as a result of a mechanical break-down, or triggered by an adversary. Consequently, the true state-transition model may deviate unpredictably from the nominal model. In the next example, we sh...

Citation Context

...nsition-uncertainty by using the notion introduced in Assumption II.2.B. Robust Dynamic Programming In this section, we present, as a basis for comparison, the algorithm and performance guarantee of =-=[10]-=-, adapted to our setting. For this section, we assume that the set of basic transition functions {P c : c ∈ C} and the modulating sequence uncertainty set D are known to the agent. For ease of exposit...

Citation Context

... have been studied before. MDPs with fixed, but unknown, reward functions and transition probabilities have been solved using robust dynamic programmingwith finite-and infinite-horizonobjectives([6], =-=[16]-=-). In general, the models for nonstationary settings limit the nonstationarity to either the rewards or the transition probabilities. MDPs where the reward functions may change arbitrarily, but the tr...

Citation Context

... random processes whose parameters change according to another process; e.g., a Markov modulated Bernoulli process is a Bernoulli process whose success probability changes according to a Markov chain =-=[5]-=-. our setting, the standard robust approach does not offer a satisfying solution. In particular, it only guarantees optimal performanceagainstthe worst realizationof theenvironment. It does not promis...