Results 1 
6 of
6
Selecting Best Practices for Effort Estimation
 IEEE Transactions on Software Engineering
, 2006
"... Abstract—Effort estimation often requires generalizing from a small number of historical projects. Generalization from such limited experience is an inherently underconstrained problem. Hence, the learned effort models can exhibit large deviations that prevent standard statistical methods (e.g., tt ..."
Abstract

Cited by 51 (22 self)
 Add to MetaCart
(Show Context)
Abstract—Effort estimation often requires generalizing from a small number of historical projects. Generalization from such limited experience is an inherently underconstrained problem. Hence, the learned effort models can exhibit large deviations that prevent standard statistical methods (e.g., ttests) from distinguishing the performance of alternative effortestimation methods. The COSEEKMO effortmodeling workbench applies a set of heuristic rejection rules to comparatively assess results from alternative models. Using these rules, and despite the presence of large deviations, COSEEKMO can rank alternative methods for generating effort models. Based on our experiments with COSEEKMO, we advise a new view on supposed “best practices ” in modelbased effort estimation: 1) Each such practice should be viewed as a candidate technique which may or may not be useful in a particular domain, and 2) tools like COSEEKMO should be used to help analysts explore and select the best method for a particular domain. Index Terms—Modelbased effort estimation, COCOMO, deviation, data mining. 1
Design and Analysis of Classifier Learning Experiments in Bioinformatics: Survey and Case
"... Abstract—In many bioinformatics applications, it is important to assess and compare the performances of algorithms trained from data, to be able to draw conclusions unaffected by chance and are therefore significant. Both the design of such experiments and the analysis of the resulting data using st ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Abstract—In many bioinformatics applications, it is important to assess and compare the performances of algorithms trained from data, to be able to draw conclusions unaffected by chance and are therefore significant. Both the design of such experiments and the analysis of the resulting data using statistical tests should be done carefully for the results to carry significance. In this paper, we first review the performance measures used in classification, the basics of experiment design and statistical tests. We then give the results of our survey over 1,500 papers published in the last two years in three bioinformatics journals (including this one). Although the basics of experiment design are well understood, such as resampling instead of using a single training set and the use of different performance metrics instead of error, only 21 percent of the papers use any statistical test for comparison. In the third part, we analyze four different scenarios which we encounter frequently in the bioinformatics literature, discussing the proper statistical methodology as well as showing an example case study for each. With the supplementary software, we hope that the guidelines we discuss will play an important role in future studies. Index Terms—Statistical tests, classification, model selection Ç
AN INCREMENTAL FRAMEWORK BASED ON CROSSVALIDATION FOR ESTIMATING THE ARCHITECTURE OF A MULTILAYER PERCEPTRON
, 2009
"... We define the problem of optimizing the architecture of a multilayer perceptron (MLP) as a state space search and propose the MOST (Multiple Operators using Statistical Tests) framework that incrementally modifies the structure and checks for improvement using crossvalidation. We consider five vari ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
We define the problem of optimizing the architecture of a multilayer perceptron (MLP) as a state space search and propose the MOST (Multiple Operators using Statistical Tests) framework that incrementally modifies the structure and checks for improvement using crossvalidation. We consider five variants that implement forward/backward search, using single/multiple operators and searching depthfirst/breadthfirst. On 44 classification and 30 regression datasets, we exhaustively search for the optimal and evaluate the goodness based on: (1) Order, the accuracy with respect to the optimal and (2) Rank, the computational complexity. We check for the effect of two resampling methods (5 × 2, tenfold cv), four statistical tests (5 × 2cvt, tenfoldcvt, Wilcoxon, sign) and two corrections for multiple comparisons (Bonferroni, Holm). We also compare with Dynamic Node Creation (DNC) and Cascade Correlation (CC). Our results show that: (1) On most datasets, networks with few hidden units are optimal, (2) forward searching finds simpler architectures, (3) variants using single node additions (deletions) generally stop early and get stuck in simple (complex) networks, (4) choosing the best of multiple operators finds networks closer to the optimal, (5) MOST variants generally find simpler networks having lower or comparable error rates than DNC and CC.
2009 International Conference on Machine Learning and Applications An Incremental Model Selection Algorithm Based on CrossValidation for Finding the Architecture of a Hidden Markov Model on Hand Gesture Data Sets
"... Abstract—In a multiparameter learning problem, besides choosing the architecture of the learner, there is the problem of finding the optimal parameters to get maximum performance. When the number of parameters to be tuned increases, it becomes infeasible to try all the parameter sets, hence we need ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract—In a multiparameter learning problem, besides choosing the architecture of the learner, there is the problem of finding the optimal parameters to get maximum performance. When the number of parameters to be tuned increases, it becomes infeasible to try all the parameter sets, hence we need an automatic mechanism to find the optimum parameter setting using computationally feasible algorithms. In this paper, we define the problem of optimizing the architecture of a Hidden Markov Model (HMM) as a state space search and propose the MSUMO (Model Selection Using Multiple Operators) framework that incrementally modifies the structure and checks for improvement using crossvalidation. There are five variants that use forward/backward search, single/multiple operators, and depthfirst/breadthfirst search. On four hand gesture data sets, we compare the performance of MSUMO with the optimal parameter set found by exhaustive search in terms of expected error and computational complexity. KeywordsHidden Markov model; model selection; crossvalidation I.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 1 Omnivariate Rule Induction Using a Novel
"... Abstract—Rule learning algorithms, for example, RIPPER, induces univariate rules, that is, a propositional condition in a rule uses only one feature. In this paper, we propose an omnivariate induction of rules where at each condition, both a univariate and a multivariate condition is trained and the ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract—Rule learning algorithms, for example, RIPPER, induces univariate rules, that is, a propositional condition in a rule uses only one feature. In this paper, we propose an omnivariate induction of rules where at each condition, both a univariate and a multivariate condition is trained and the best is chosen according to a novel statistical test. This paper has three main contributions: First, we propose a novel statistical test, the combined 5×2 cvt test, to compare two classifiers, which is a variantofthe5×2 cvttest and give the connections to other tests as 5 × 2 cv F test and kfold paired t test. Second, we propose a multivariate version of RIPPER where Support Vector Machine (SVM) with linear kernel is used to find multivariate linear conditions. Third, we propose an omnivariate version of RIPPER where the model selection is done via the combined 5×2 cv t test. Our results indicate that (1) the combined 5×2 cvt test has higher power (lower type II error), lower type I error, and higher replicability compared to the 5×2cvttest, (2) omnivariate rules are better in that they choose whichever condition is more accurate, selecting the right model automatically and separately for each condition in a rule. Index Terms—rule induction, model selection, statistical tests, support vector machines I.
unknown title
"... � Is the error rate of my classifier less than 2%? � Is kNN more accurate than MLP? � Does having PCA before improve accuracy? ..."
Abstract
 Add to MetaCart
� Is the error rate of my classifier less than 2%? � Is kNN more accurate than MLP? � Does having PCA before improve accuracy?