Results 1  10
of
33
The control of the false discovery rate in multiple testing under dependency
 Annals of Statistics
, 2001
"... Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparab ..."
Abstract

Cited by 469 (8 self)
 Add to MetaCart
Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate t. Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which
Statistical Comparisons of Classifiers over Multiple Data Sets
, 2006
"... While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but igno ..."
Abstract

Cited by 243 (0 self)
 Add to MetaCart
While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust nonparametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding posthoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.
Adapting to unknown sparsity by controlling the false discovery rate
, 2000
"... We attempt to recover a highdimensional vector observed in white noise, where the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing powerlaw decay bounds on the order ..."
Abstract

Cited by 108 (15 self)
 Add to MetaCart
We attempt to recover a highdimensional vector observed in white noise, where the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing powerlaw decay bounds on the ordered entries; and controlling the ℓp norm for p small. We obtain a procedure which is asymptotically minimax for ℓr loss, simultaneously throughout a range of such sparsity classes. The optimal procedure is a dataadaptive thresholding scheme, driven by control of the False Discovery Rate (FDR). FDR control is a recent innovation in simultaneous testing, in which one seeks to ensure that at most a certain fraction of the rejected null hypotheses will correspond to false rejections. In our treatment, the FDR control parameter q also plays a controlling role in asymptotic minimaxity. Our results say that letting q = qn → 0 with problem size n is sufficient for asymptotic minimaxity, while keeping fixed q>1/2prevents asymptotic minimaxity. To our knowledge, this relation between ideas in simultaneous inference and asymptotic decision theory is new. Our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors. These new rules impose complexity penalization of the form 2·log ( potential model size / actual model size). We exhibit a close connection with FDRcontrolling procedures having q tending to 0; this connection strongly supports a conjecture of simultaneous asymptotic minimaxity for such model selection rules.
An extension on ―statistical comparisons of classifiers over multiple data sets‖ for all pairwise comparisons
 Journal of Machine Learning Research
"... In a recently published paper in JMLR, Demˇsar (2006) recommends a set of nonparametric statistical tests and procedures which can be safely used for comparing the performance of classifiers over multiple data sets. After studying the paper, we realize that the paper correctly introduces the basic ..."
Abstract

Cited by 54 (13 self)
 Add to MetaCart
In a recently published paper in JMLR, Demˇsar (2006) recommends a set of nonparametric statistical tests and procedures which can be safely used for comparing the performance of classifiers over multiple data sets. After studying the paper, we realize that the paper correctly introduces the basic procedures and some of the most advanced ones when comparing a control method. However, it does not deal with some advanced topics in depth. Regarding these topics, we focus on more powerful proposals of statistical procedures for comparing n×n classifiers. Moreover, we illustrate an easy way of obtaining adjusted and comparable pvalues in multiple comparison procedures.
Testing Efficient Risk Sharing with Heterogeneous Risk Preferences ∗
"... Previous papers have tested efficient risk sharing under the assumption of identical risk preferences. In this paper we show that, if in the data households have heterogeneous risk preferences, the tests proposed in the past reject efficiency even if households share risk efficiently. To address thi ..."
Abstract

Cited by 15 (0 self)
 Add to MetaCart
Previous papers have tested efficient risk sharing under the assumption of identical risk preferences. In this paper we show that, if in the data households have heterogeneous risk preferences, the tests proposed in the past reject efficiency even if households share risk efficiently. To address this issue we propose a method that enables one to test efficiency even when households have different preferences for risk. The method is composed of three tests. The first one can be used to determine whether in the data under investigation households have homogeneous risk preferences. The second and third test can be used to evaluate efficient risk sharing when the hypothesis of homogeneous risk preferences is rejected. We use this method to test efficient risk sharing in rural India. Using the first test, we strongly reject the hypothesis of identical risk preferences. We then test efficiency with and without the assumption of preference homogeneity. In the first case we reject efficient risk sharing at the village and caste level. In the second case we still reject efficiency at the village level, but we cannot reject this hypothesis at the caste level. This finding suggests that the relevant risksharing unit in rural India is the caste and not the village. 1
Abstract Histone Acetylation and Transcriptional Regulation in the Genome of Saccharomyces cerevisiae
, 2005
"... Motivation: In eukaryotic genomes, histone acetylation and thereafter departure from the chromatin is essential for gene transcription initiation. Because gene transcription is tightly regulated by transcription factors, there are some speculations on the cooperation of histone acetylation and trans ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
Motivation: In eukaryotic genomes, histone acetylation and thereafter departure from the chromatin is essential for gene transcription initiation. Because gene transcription is tightly regulated by transcription factors, there are some speculations on the cooperation of histone acetylation and transcription factor binding. However, systematic statistical analyses of this relationship on a genomic scale have not been reported. Results: We apply several statistical methods to explore this relationship on two recent genomic data sets: acetylation levels on 11 histone lysines and binding activities of 203 transcription factors, both in promoter regions across the yeast genome. By canonical correlation analysis, we find that a histone acetylation pattern is correlated with certain profile of transcription factor binding in the genome. Furthermore, after clustering the genes by their acetylation levels on the 11 histone lysines, the genes within clusters show distinct transcription factor binding profiles, as discovered by principle component analysis. Even after applying fairly stringent statistical measurement, most of these clusters have transcription factors with binding activities significantly deviated from the overall genome. We conclude that in the yeast genome, there is a strong correlation between histone acetylation and transcription factor binding in the promoter regions.
RECENT ADVANCES IN MULTIPLE TESTING
"... There has been renewed interest in the area of multiple testing because of its importance in many statistical investigations, particularly in experiments where large data sets are generated, such as genetic microarrays. The purpose of this paper is to present some important developments that have ta ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
There has been renewed interest in the area of multiple testing because of its importance in many statistical investigations, particularly in experiments where large data sets are generated, such as genetic microarrays. The purpose of this paper is to present some important developments that have taken place recently in this area, focusing mainly on stepwise multiple testing procedures, and to introduce some open problems.
www.elsevier.com/locate/jspi A twostep rejection procedure for testing multiple hypotheses
, 2007
"... This paper considers pvalue based stepwise rejection procedures for testing multiple hypotheses. The existing procedures have used constants as critical values at all steps. With the intention of incorporating the exact magnitude of the pvalues at the earlier steps into the decisions at the later ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
This paper considers pvalue based stepwise rejection procedures for testing multiple hypotheses. The existing procedures have used constants as critical values at all steps. With the intention of incorporating the exact magnitude of the pvalues at the earlier steps into the decisions at the later steps, this paper applies a different strategy that the critical values at the later steps are determined as functions of the pvalues from the earlier steps. As a result, we have derived a new equality and developed a twostep rejection procedure following that. The new procedure is a shortcut of a stepup procedure, and it possesses great simplicity. In terms of power, the proposed procedure is generally comparable to the existing ones and exceptionally superior when the largest pvalue is anticipated to be less than 0.5.
A MULTISTAGE MULTIPLE COMPARISON PROCEDURE FOR THE ANALYSIS OF MULTIPLE TREATMENT GROUP CLINICAL TRIALS
, 1994
"... Multiple treatment group clinical trials are frequently used in Phase III clinical drug development to establish differences between the treatment groups for the purposes of satisfying regulatory requirements. Because of the multiplicity of treatment group comparisons, the design and analysis should ..."
Abstract
 Add to MetaCart
Multiple treatment group clinical trials are frequently used in Phase III clinical drug development to establish differences between the treatment groups for the purposes of satisfying regulatory requirements. Because of the multiplicity of treatment group comparisons, the design and analysis should consider a multiple comparison procedure which controls the experimentwise Type I error rate a and maintains power. The procedure should also be suitable for the design objectives of the trial. Many common multiple comparison procedures (MCPs) control the experimentwise Type I error rate in a liberal way, while other MCPs control the experimentwise Type I error rate in a conservative way. Generally, liberal MCPs are more powerful than conservative MCPs, however, do not control the experimentwise Type I error rate at the nominal level. Many conservative MCPs may lack power to satisfy study
Some Remarks on Simes type multiple tests of significance
, 1997
"... Simes ' theorem, posed in a multiple testing setup, has its genesis in the classical ballot theorem in uniform order statistics. The purpose of this note is to comment on this rediscovery of the Ballot theorem in multiple testing theory, and with a view to enhance scope of applications, to stress it ..."
Abstract
 Add to MetaCart
Simes ' theorem, posed in a multiple testing setup, has its genesis in the classical ballot theorem in uniform order statistics. The purpose of this note is to comment on this rediscovery of the Ballot theorem in multiple testing theory, and with a view to enhance scope of applications, to stress its ramifications in the same context. This privides sharper error rates. AMS Subject Classifications: 62J15, 62E15.