## Bias plus variance decomposition for zero-one loss functions (1996)

### Cached

### Download Links

- [robotics.stanford.edu]
- [ai.stanford.edu]
- [robotics.stanford.edu]
- [robotics.stanford.edu]
- DBLP

### Other Repositories/Bibliography

Venue: | In Machine Learning: Proceedings of the Thirteenth International Conference |

Citations: | 187 - 5 self |

### BibTeX

@INPROCEEDINGS{Kohavi96biasplus,

author = {Ron Kohavi},

title = {Bias plus variance decomposition for zero-one loss functions},

booktitle = {In Machine Learning: Proceedings of the Thirteenth International Conference},

year = {1996},

pages = {275--283}

}

### Years of Citing Articles

### OpenURL

### Abstract

We present a bias-variance decomposition of expected misclassi cation rate, the most commonly used loss function in supervised classi cation learning. The bias-variance decomposition for quadratic loss functions is well known and serves as an important tool for analyzing learning algorithms, yet no decomposition was o ered for the more commonly used zero-one (misclassi cation) loss functions until the recent work of Kong & Dietterich (1995) and Breiman (1996). Their decomposition su ers from some major shortcomings though (e.g., potentially negative variance), which our decomposition avoids. We show that, in practice, the naive frequency-based estimation of the decomposition terms is by itself biased and show how to correct for this bias. We illustrate the decomposition on various algorithms and datasets from the UCI repository. 1

### Citations

9216 |
Elements of Information Theory
- Cover, Thomas
- 1991
(Show Context)
Citation Context ...en averaging, minus estimating based on the full set of N =2n training sets): E 1 P 2 i=1�2 (T ; wi) 2 ; T ; 1 2 P i=1�2 wi 2 = 1 P 2 i=1�2 E (T ; wi) 2 ; E T ; 1 P wi 2 i=1�2 By Jensen's inequality (=-=Cover & Thomas 1991-=-), the rst term on the right hand side is larger or equal to the second term. This shows that when we average once (over 2n instances) rather than twice (over n instances) we get a smaller estimate fo... |

3642 | Induction of Decision Trees
- Quinlan
- 1986
(Show Context)
Citation Context ...set E. (Equation 4 was used to estimate p(yHjf� m� x).) At rst, all these terms were estimated using frequency counts. Figure 1 (left) shows the estimate for bias 2 for di erent values of N when ID3 (=-=Quinlan 1986-=-) was executed on three datasets from the UCI repository. It is clear that our estimate of bias 2 using frequency counts shrinks as we increase N. Since in nite N gives the correct value of bias 2 , t... |

3362 | An introduction to the bootstrap - Efron, Tibshirani - 1993 |

2761 | Bagging predictors
- Breiman
- 1996
(Show Context)
Citation Context ...ng Classi ers There has been a lot of work recently on combining classi ers, with the terms aggregation, averages, ensembles, classi er combinations, voting, and stacking commonly used (Wolpert 1992, =-=Breiman 1994-=-a, Perrone 1993, Ali 1996). In the simplest scheme, multiple classi ers are generated and then vote for each test instance, with the majority prediction used as the nal prediction. Figure 3 shows ID3 ... |

787 |
Uci repository of machine learning databases, machine-readable data repository
- Murphy, Aha
- 1996
(Show Context)
Citation Context ...erms in our decomposition by using frequency counts. 4.1 Our frequency counts experiments To investigate the behavior of the terms in our decomposition, we ran a set of experiments on UCI repository (=-=Murphy & Aha 1996-=-). In each of those experiments, for a given dataset and a given learning algorithm, we estimated (the x-average of) bias 2 ,variance, intrinsic noise, and overall error as follows. 1. We randomly div... |

646 | Neural Networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992 |

580 | Stacked generalization
- Wolpert
- 1992
(Show Context)
Citation Context ...n. 4.5 Combining Classi ers There has been a lot of work recently on combining classi ers, with the terms aggregation, averages, ensembles, classi er combinations, voting, and stacking commonly used (=-=Wolpert 1992-=-, Breiman 1994a, Perrone 1993, Ali 1996). In the simplest scheme, multiple classi ers are generated and then vote for each test instance, with the majority prediction used as the nal prediction. Figur... |

155 | Error-correcting output coding corrects bias and variance - Kong, Dietterich - 1995 |

98 | variance and arcing classifiers - Breiman |

81 | Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization, Unpublished doctoral thesis
- Perrone
- 1993
(Show Context)
Citation Context ...here has been a lot of work recently on combining classi ers, with the terms aggregation, averages, ensembles, classi er combinations, voting, and stacking commonly used (Wolpert 1992, Breiman 1994a, =-=Perrone 1993-=-, Ali 1996). In the simplest scheme, multiple classi ers are generated and then vote for each test instance, with the majority prediction used as the nal prediction. Figure 3 shows ID3 versus a combin... |

51 | The heuristics of instability in model selection - Breiman - 1996 |

50 | Machine learning bias, statistical bias, and statistical variance of decision tree algorithms - Dietterich, Kong - 1995 |

23 |
Learning probabilistic Relational Concept Descriptions
- Ali
- 1996
(Show Context)
Citation Context ...a lot of work recently on combining classi ers, with the terms aggregation, averages, ensembles, classi er combinations, voting, and stacking commonly used (Wolpert 1992, Breiman 1994a, Perrone 1993, =-=Ali 1996-=-). In the simplest scheme, multiple classi ers are generated and then vote for each test instance, with the majority prediction used as the nal prediction. Figure 3 shows ID3 versus a combination of 5... |

18 |
Almost sure consistent nonparametric regression from recursive partitioning schemes. Journal of Multivariate Analysis 15:147{163
- Gordon, Olshen
- 1984
(Show Context)
Citation Context ...equence, decision trees are very unstable, and therefore they usually gain by aggregation techniques (Breiman 1994b). Note also that the smaller the internal sample, the more bias we potentially add (=-=Gordon & Olshen 1984-=-) but the more di erent the classi ers will be, leading to a more stable average. Our results show that in this voting scheme, the reduction in error is almost solely due to the reduction in variance.... |

3 | Bias, variance, and arcing classi ers - Breiman - 1996 |

1 |
Heuristics of instability inmodel selection
- Breiman
- 1994
(Show Context)
Citation Context ...ng Classi ers There has been a lot of work recently on combining classi ers, with the terms aggregation, averages, ensembles, classi er combinations, voting, and stacking commonly used (Wolpert 1992, =-=Breiman 1994-=-a, Perrone 1993, Ali 1996). In the simplest scheme, multiple classi ers are generated and then vote for each test instance, with the majority prediction used as the nal prediction. Figure 3 shows ID3 ... |