Citation Context

...finite number of features and the reward function is linear in these features. Other authors have studied cases where the reward function is Lipschitz continuous [17], sampled from a Gaussian process =-=[25]-=-, or takes the form of a generalized [13] or sparse [3] linear model. The first author is supported by a Burt and Deedee McMurty Stanford Graduate Fellowship. This work was supported in part by Award ...

Citation Context

...lynomial dependence on the dimension. Moreover, our performance guarantees depend explicitly on what we could have achieved if we had known the subspace in advance. Previous work. Exploration–exploitation tradeoffs were originally studied in the context of finite multi-armed bandits [8]. Since then, results have been obtained for continuous domains, starting with the linear [9] and Lipschitz-continuous cases [10, 11]. A more recent algorithm that enjoys theoretical bounds for functions sampled from a Gaussian Process (GP), or belong to some Repro1 ducible Kernel Hilbert Space (RKHS) is GP-UCB [12]. The use of GPs to negotiate exploration– exploitation tradeoffs originated in the areas of response surface and Bayesian optimization, for which there are a number of approaches (cf., [13]), perhaps most notably the Expected Improvement [14] approach, which has recently received theoretical justification [15], albeit only in the noise-free setting. Bandit algorithms that exploit low-dimensional structure of the function appeared first for the linear setting, where under sparsity assumptions one can obtain bounds, which depend only weakly on the ambient dimension [16, 17]. In [18] the more ge...

Citation Context

... ( ‖Sτ‖2V̄ −1τ > 2R2 log ( det ( I +M1:tV −1M∗1:t )1/2 δ )) ≤ δ . 4That is, for any x, y ∈ H, 〈x, U−1s y〉 → 〈x, V −1y〉. 20 The quantity log det ( I +M1:τV −1M∗1:τ ) is bounded for several kernels in (=-=Srinivas et al., 2012-=-). Here, we demonstrate a simple bound for two special cases. Corollary 3.7 Assume that the vectors (mk) come from a finite, K-element set, {v1, . . . , vK} ⊂ H. Then log det ( I +M1:τV −1M∗1:τ ) ≤ K ...

Citation Context

...finite X , we have rKt ď rp0qt ď 2 ? βtσ 0 t , and for compact and convex X following the assumptions of Theorem 1, rKt ď rp0qt ď 2 ? βtσ 0 t ` 1t2 , holds with probability at least 1´ δ. We refer to =-=[6]-=- (Lemmas 5.2, 5.8) for the detailed proof of the bound for r p0q t . Now we show an intermediate result bounding the deviations at the points x0t`1 by the one at the points x K´1 t . Lemma 2. The devi...

Citation Context

...ies broadly, beyond the scope of specific problems that have been studied in the literature, and also identifies fundamental insights that unify more specialized prior results. The guarantees we provide apply to two popular classes of algorithms for the stochastic MAB: upper confidence bound (UCB) algorithms and Thompson sampling. Each algorithm is described in Section 3. The aforementioned papers on the linear bandit problem study UCB algorithms [1– 3]. Other authors have studied UCB algorithms in cases where the reward function is Lipschitz continuous [4, 5], sampled from a Gaussian process [6], or takes the form of a generalized [7] or sparse [8] linear model. More generally, there is an immense literature on this approach to balancing between exploration and exploitation, including work on bandits with independent arms [9–12], reinforcement learning [13, 14], and Monte Carlo Tree Search [15]. Recently, a simple posterior sampling algorithm called Thompson sampling was shown to share a close connection with UCB algorithms [16]. This connection enables us to study both types of algorithms in a unified manner. Though it was first proposed in 1933 [17], Thompson sampling has until rec...

Citation Context

...ce guarantees in the form of regret bounds. Surprisingly, virtually all of these regret bounds depend on hard knowledge but not soft knowledge, a notable exception being the bounds of Srinivas et al. =-=[28]-=- which we discuss further in Section 1.2. Regret bounds that depend on hard knowledge yield insight into how an algorithm’s performance scales with the complexity of the family of mappings and are use...

Citation Context

...t al. [3] developed upper confidence bound (UCB) algorithms for multiarmed bandits with bounded reward that achieve logarithmic expected cumulative regret uniformly in time. Recently, Srinivas et al. =-=[4]-=- developed asymptotically optimal UCB algorithms for Gaussian process optimization. Kauffman et al. [5] developed a generic Bayesian UCB algorithm and established its optimality for binary bandits wit...