## An Effective Method for Correlated Selection Problems (1994)

Citations: | 4 - 2 self |

### BibTeX

@TECHREPORT{Gratch94aneffective,

author = {Jonathan Gratch},

title = {An Effective Method for Correlated Selection Problems},

institution = {},

year = {1994}

}

### OpenURL

### Abstract

This article is organized as follows. The next selection reviews the standard statistical approaches to selection problems and introduce some of the terminology that has been developed in this area. Section 3 discusses the method of multiple comparisons and illustrate how this method can be applied to solving correlated selection problems. From this perspective, a selection problem reduces to the problem of simultaneously performing a number of two-hypothesis selection problems. Section 4 describes the Sequential Probability Ratio Test (SPRT), an efficient method for solving a two-hypothesis selection problem. Section 5 describes a new correlated selection method, MC-SPRT, which combines the multiple comparison approach with the SPRT. Section 6 shows how the cost of selection can be further improved by a decision-theoretic evaluation of the cost of taking examples.

### Citations

1245 |
Statistical decision theory and Bayesian analysis
- Berger
- 1985
(Show Context)
Citation Context ...chhofer54]; also known as probably approximately correct (PAC) selection in machine learning circles. Bayesian approaches to selection problems typically bound the expected loss of making a selection =-=[Berger80]-=-, where loss is some pre-specified function of the hypotheses. For example, a common loss function weights the probability of an incorrect selection by how wrong the selection is [Chien95]. Some appro... |

440 |
Sequential Analysis
- Wald
- 1947
(Show Context)
Citation Context ...wn to have error less than or equal to PCE on the original problem when |q|.e.sThe simpler problem can be solved optimally using a statistical test called the sequential probability ratio test (SPRT) =-=[Wald47]-=-.sSPRT is optimal in the sense that there is no other statistical test with at least as low probability of error and with smaller expected sample sizes.sThis optimality property, however, does not nec... |

175 |
Introduction to Mathematical Statistics
- HOGG, T
- 1965
(Show Context)
Citation Context ...r example, the Cauchy distribution. (1) (2) 4 theorem demonstrates that even if the underlying distribution is non-normal, the assumption of normality is reasonable when estimating its expected value =-=[Hogg78]. As-=-suming normality means a utility distribution has the following form: u(x) = 1 # 2# # exp#-- (x -- #) 2 2# 2 # where �� is the expected utility of the hypothesis, s 2 is the variance or dispersion... |

128 | Efficient algorithms for minimizing cross validation error
- Moore, Lee
- 1994
(Show Context)
Citation Context ...ing concept description [Fayyad91, Musick93]; the feature selection problem consists of selecting one of a set of feature vectors to learn from where there is considerable overlap between the vectors =-=[Moore94]-=-. In all of these problems the hypotheses have considerable common structure and therefore their performance on data will tend to be highly positively correlated. This article is organized as follows.... |

101 | Hoeffding races: Accelerating model selection search for classification and function approximation - Maron, Moore - 1994 |

69 |
A single-sample multiple decision procedure for ranking means of normal populations with known variances
- Bechhofer
- 1954
(Show Context)
Citation Context ... A less restrictive assertion is that the selected hypothesis is close to the best with some confidence. This later assertion leads to a class of selection problems called indifference-zone selections=-=[Bechhofer54]-=-; also known as probably approximately correct (PAC) selection in machine learning circles. Bayesian approaches to selection problems typically bound the expected loss of making a selection [Berger80]... |

69 |
COMPOSER: A probabilistic solution to the utility problem in speed-up learning
- Gratch, Dejong
- 1992
(Show Context)
Citation Context ...g the sign with error no more than PCE. Efficient methods for this problem include the repeated significance test (RST) [Lerche86] and the N��das approach used in our earlier solution to this prob=-=lem [Gratch92]-=-. An undesirable property of these methods, however is that their sample complexity tends to infinity as the expected difference approaches zero. Instead, we introduce an indifference parameter, e, th... |

46 | A statistical approach to solving the EBL utility problem - Greiner, Jurisica - 1992 |

41 | Decision theoretic subsampling for induction on large databases - Musick, Catlett, et al. - 1993 |

38 | On the induction of decision trees for multiple concept learning - Fayyad - 1991 |

24 |
A sequential procedure for selecting the population with the largest mean from k normal populations
- Paulson
(Show Context)
Citation Context ... for some selection problems, I suggest ways to construct a closed procedure later in the article). The standard approach to sequential unbalanced allocation is what are called elimination algorithms =-=[Paulson64]-=-. Elimination algorithms have characteristic that they eliminate inferior hypotheses as early as possible, during the course of selection. As, inferior hypotheses will tend to have fewer examples allo... |

18 |
On the efficient allocation of resources for hypothesis evaluation: A statistical approach
- Chien, Gratch, et al.
- 1995
(Show Context)
Citation Context ... selection [Berger80], where loss is some pre-specified function of the hypotheses. For example, a common loss function weights the probability of an incorrect selection by how wrong the selection is =-=[Chien95]-=-. Some approaches make selections based on other characteristics of hypotheses besides their expected utility. For example, Satner and Tamhane propose a procedure that selects a hypothesis with large ... |

11 |
Using Common Random Numbers and Control Variates in MultipleComparison Procedures”. Operations Research 39 (4): 583–591
- Yang, Nelson
- 1991
(Show Context)
Citation Context ...se of correlated hypotheses, a technique called blocking can be used to reduce this variance and thus reduce the sample complexity (this approach is also called the use of common random numbers (CRN) =-=[Yang91]-=-). Blocking rests on the observation that there are two sources of variation in a selection problem: differences in the hypotheses and differences in the data. When hypotheses are positively cor6 rela... |

10 | A Bonferroni selection procedure when using common random numbers with unknown variances - Clark, Yang - 1986 |

8 |
The Sequential Statistical Analysis
- Govindarajulu
- 1981
(Show Context)
Citation Context ...design tradeoffs can be made. Selection techniques differ in how examples are allocated to hypotheses in order to estimate expected utility. One distinction is between fixed and sequential strategies =-=[Govindarajulu81]-=-. Fixed strategies determine in advance howmuch data to allocate to each hypothesis, process all of the data, and then make a selection. In contrast, sequential strategies take data a block at a time,... |

8 | Improving Learning Performance Through Rational Resource Allocation
- Gratch, Chien, et al.
- 1994
(Show Context)
Citation Context ...process them are considered valuable resources. A rational allocation policy is one that allocates observations to hypotheses in such a way as to minimize the expected cost of making a selection (see =-=[Gratch94]-=-). This expected cost depends on the expected number of observations allocated to each hypotheses as well as the expected cost to obtain each of those observations. The cost of obtaining a utility obs... |

7 |
Theories of data analysis: From magical thinking through classical statistics
- Diaconis
- 1985
(Show Context)
Citation Context ...orming multiple comparisons, a statistical test must account for the multiplicity effect. This is the observation that "if enough statistics are computed, some of them will be sure to show struct=-=ure" [Diaconis85]-=-. The principle contribution of multiple comparison methods is a body of techniques for assigning significance levels to the pair-wise comparisons such that some overall significance level is reached.... |

1 |
A modification of the sequential probability ration test to reduce the sample size
- Anderson
- 1960
(Show Context)
Citation Context ...nder several parameter settings. 4.3 Alternatives to the SPRT SPRT is optimal for the problem of testing q=-e versus q=e, but it is not necessarily optimal for the original problem of q3-e versus q.e =-=[Anderson60]-=- (in fact, it is not even clear what optimality means in this context). In particular, a disadvantage of SPRT is that the expected sample size is large for values of q within the indifference-zone. Th... |

1 |
The effect of truncation on a sequential test for the drift of brownian motion
- Bather, Chernoff, et al.
- 1989
(Show Context)
Citation Context ... appropriately set the stopping boundary.sBather and Chernoff describe a Bayesian non-SPRT approach for efficiently testing the sign of a mean by applying truncation to the repeated significance test =-=[Bather89]-=-. 5. MULTIPLE COMPARISON SPRT (MC-SPRT) We can now summarize our multiple comparison SPRT (MC-SPRT) approach to solving correlated selection problems. We first describe the generic approach and then d... |

1 |
Extended--Paulson Sequential Selection
- Edwards
- 1987
(Show Context)
Citation Context ...attention to classical indifference-zone selection. In particular we will focus on two selection assertions that have been studied extensively in this literature. The two assertions are (from Edwards =-=[Edwards87]-=-): Assertion 2.1 (Indifference Zone). The selected hypothesis is the e-best hypothesis, if there is only one e-best hypothesis. Assertion 2.2 (Confidence Bound). The selected hypothesis is a e-best hy... |

1 | Sequential Multiple Comparisons With the Best - Hsu, Edwards - 1983 |

1 |
Tamhane, "Multiple Comparison Procedures
- Lerche, Hochberg, et al.
- 1987
(Show Context)
Citation Context ...tive tack on this problem [Nelson95]). The method of multiple comparisons is an approach for simultaneously performing a number of pairwise statistical comparisons between hypotheses drawn from a set =-=[Lerche86]-=-. For example, to detect differences in the means of k . 3 hypotheses, the method of multiple comparisons would suggest performingsk choose 2 separate pair-wise t-tests. When performing multiple compa... |

1 |
A Note on the Variance and Higher Central Moments of the Stopping Time of an SPRT
- Martinsek
- 1981
(Show Context)
Citation Context ...E # S N # 1 # # [a + (b -- a)# ~ (#)] = E ~ # N Case 2. # # = 0 : from Theorem 11(ii) E # S N # -- ab # 2 # = E ~ # N Martinsek's Approximation to the Variance in the Expected Stopping Time Martinsek =-=[Martinsek81] der-=-ives the asymptotic behavior of var(N) as the stopping boundary, b, approaches infinity. Empirical results suggest that this is a reasonable approximation for finite values of b provided that �� q... |

1 |
Using Common Random Numbers for Indifference
- Nelson, Natejcik
- 1995
(Show Context)
Citation Context ...hese techniques, a selection problem reduces to the problem of simultaneously performing a number of two-hypothesis selection problems (see Nelson and Matejcik for an alternative tack on this problem =-=[Nelson95]-=-). The method of multiple comparisons is an approach for simultaneously performing a number of pairwise statistical comparisons between hypotheses drawn from a set [Lerche86]. For example, to detect d... |

1 |
Tamhane, "Designing Experiments for Selecting a Normal Population with a Large
- Santner, C
- 1984
(Show Context)
Citation Context ...sed on other characteristics of hypotheses besides their expected utility. For example, Satner and Tamhane propose a procedure that selects a hypothesis with large expected utility and small variance =-=[Santner84]-=-. In this article we will restrict attention to classical indifference-zone selection. In particular we will focus on two selection assertions that have been studied extensively in this literature. Th... |