Results 1 
9 of
9
Automatic Construction of Decision Trees from Data: A MultiDisciplinary Survey
 Data Mining and Knowledge Discovery
, 1997
"... Decision trees have proved to be valuable tools for the description, classification and generalization of data. Work on constructing decision trees from data exists in multiple disciplines such as statistics, pattern recognition, decision theory, signal processing, machine learning and artificial ne ..."
Abstract

Cited by 147 (1 self)
 Add to MetaCart
Decision trees have proved to be valuable tools for the description, classification and generalization of data. Work on constructing decision trees from data exists in multiple disciplines such as statistics, pattern recognition, decision theory, signal processing, machine learning and artificial neural networks. Researchers in these disciplines, sometimes working on quite different problems, identified similar issues and heuristics for decision tree construction. This paper surveys existing work on decision tree construction, attempting to identify the important issues involved, directions the work has taken and the current state of the art. Keywords: classification, treestructured classifiers, data compaction 1. Introduction Advances in data collection methods, storage and processing technology are providing a unique challenge and opportunity for automated data exploration techniques. Enormous amounts of data are being collected daily from major scientific projects e.g., Human Genome...
RainForest  a Framework for Fast Decision Tree Construction of Large Datasets
 In VLDB
, 1998
"... Classification of large datasets is an important data mining problem. Many classification algorithms have been proposed in the literature, but studies have shown that so far no algorithm uniformly outperforms all other algorithms in terms of quality. In this paper, we present a unifying framework fo ..."
Abstract

Cited by 95 (9 self)
 Add to MetaCart
Classification of large datasets is an important data mining problem. Many classification algorithms have been proposed in the literature, but studies have shown that so far no algorithm uniformly outperforms all other algorithms in terms of quality. In this paper, we present a unifying framework for decision tree classifiers that separates the scalability aspects of algorithms for constructing a decision tree from the central features that determine the quality of the tree. This generic algorithm is easy to instantiate with specific algorithms from the literature (including C4.5, CART,
Lookahead and Pathology in Decision Tree Induction
 Proceedings of the 14th International Joint Conference on Artificial Intelligence
, 1995
"... The standard approach to decision tree induction is a topdown, greedy algorithm that makes locally optimal, irrevocable decisions at each node of a tree. In this paper, we study an alternative approach, in which the algorithms use limited lookahead to decide what test to use at a node. We systemati ..."
Abstract

Cited by 52 (2 self)
 Add to MetaCart
The standard approach to decision tree induction is a topdown, greedy algorithm that makes locally optimal, irrevocable decisions at each node of a tree. In this paper, we study an alternative approach, in which the algorithms use limited lookahead to decide what test to use at a node. We systematically compare, using a very large number of decision trees, the quality of decision trees induced by the greedy approach to that of trees induced using lookahead. The main results of our experiments are: (i) the greedy approach produces trees that are just as accurate as trees produced with the much more expensive lookahead step; and (ii) decision tree induction exhibits pathology, in the sense that lookahead can produce trees that are both larger and less accurate than trees produced without it. 1. Introduction The standard algorithm for constructing decision trees from a set of examples is greedy induction  a tree is induced topdown with locally optimal choices made at each node, with...
Lookaheadbased Algorithms for Anytime Induction of Decision Trees
 In ICML’04
, 2004
"... The majority of the existing algorithms for learning decision trees are greedya tree is induced topdown, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Furthermore, the greedy algorithms require a fixed amount of time and are no ..."
Abstract

Cited by 13 (2 self)
 Add to MetaCart
The majority of the existing algorithms for learning decision trees are greedya tree is induced topdown, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Furthermore, the greedy algorithms require a fixed amount of time and are not able to generate a better tree if additional time is available. To overcome this problem, we present two lookaheadbased algorithms for anytime induction of decision trees, thus allowing tradeoff between tree quality and learning time. The first one is depthk lookahead, where a larger time allocation permits larger k. The second algorithm uses a novel strategy for evaluating candidate splits; a stochastic version of ID3 is repeatedly invoked to estimate the size of the tree in which each split results, and the one that minimizes the expected size is preferred. Experimental results indicate that for several hard concepts, our proposed approach exhibits good anytime behavior and yields significantly better decision trees when more time is available.
Anytime learning of decision trees
 Journal of Machine Learning Research
"... The majority of existing algorithms for learning decision trees are greedy—a tree is induced topdown, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Even the few nongreedy learners cannot learn good trees when the concept is diff ..."
Abstract

Cited by 8 (3 self)
 Add to MetaCart
The majority of existing algorithms for learning decision trees are greedy—a tree is induced topdown, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Even the few nongreedy learners cannot learn good trees when the concept is difficult. Furthermore, they require a fixed amount of time and are not able to generate a better tree if additional time is available. We introduce a framework for anytime induction of decision trees that overcomes these problems by trading computation speed for better tree quality. Our proposed family of algorithms employs a novel strategy for evaluating candidate splits. A biased sampling of the space of consistent trees rooted at an attribute is used to estimate the size of the minimal tree under that attribute, and an attribute with the smallest expected tree is selected. We present two types of anytime induction algorithms: a contract algorithm that determines the sample size on the basis of a pregiven allocation of time, and an interruptible algorithm that starts with a greedy tree and continuously improves subtrees by additional sampling. Experimental results indicate that, for several hard concepts, our proposed approach exhibits good anytime behavior and yields significantly better decision trees when more time is available.
Decision Tree Induction: How Effective is the Greedy Heuristic?
 In Proceedings of the First International Conference on Knowledge Discovery and Data Mining
, 1995
"... Most existing decision tree systems use a greedy approach to induce trees  locally optimal splits are induced at every node of the tree. Although the greedy approach is suboptimal, it is believed to produce reasonably good trees. In the current work, we attempt to verify this belief. We quantify ..."
Abstract

Cited by 8 (4 self)
 Add to MetaCart
Most existing decision tree systems use a greedy approach to induce trees  locally optimal splits are induced at every node of the tree. Although the greedy approach is suboptimal, it is believed to produce reasonably good trees. In the current work, we attempt to verify this belief. We quantify the goodness of greedy tree induction empirically, using the popular decision tree algorithms, C4.5 and CART. We induce decision trees on thousands of synthetic data sets and compare them to the corresponding optimal trees, which in turn are found using a novel map coloring idea. We measure the effect on greedy induction of variables such as the underlying concept complexity, training set size, noise and dimensionality. Our experiments show, among other things, that the expected classification cost of a greedily induced tree is consistently very close to that of the optimal tree. Introduction Decision trees are known to be effective classifiers in a variety of domains. Most of the methods ...
Interruptible anytime algorithms for iterative improvement of decision trees
 In Proceedings of Workshop on the UtilityBased Data Mining (UBDM2005), held with The 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'05
, 2005
"... Finding a minimal decision tree consistent with the examples is an NPcomplete problem. Therefore, most of the existing algorithms for decision tree induction use a greedy approach based on local heuristics. These algorithms usually require a fixed small amount of time and result in trees that are n ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
Finding a minimal decision tree consistent with the examples is an NPcomplete problem. Therefore, most of the existing algorithms for decision tree induction use a greedy approach based on local heuristics. These algorithms usually require a fixed small amount of time and result in trees that are not globally optimal. Recently, the LSID3 contract anytime algorithm was introduced to allow using extra resources for building better decision trees. A contract anytime algorithm needs to get its resource allocation a priori. In many cases, however, the time allocation is not known in advance, disallowing the use of contract algorithms. To overcome this problem, in this work we present two interruptible anytime algorithms for inducing decision trees. Interruptible anytime algorithms do not require their resource allocation in advance and thus must be ready to be interrupted and return a valid solution at any moment. The first interruptible algorithm we propose is based on a general technique for converting a contract algorithm to an interruptible one by sequencing. The second is an iterative improvement algorithm that repeatedly selects a subtree whose reconstruction is estimated to yield the highest marginal utility and rebuilds it with higher resource allocation. Empirical evaluation shows a good anytime behavior for both algorithms. The iterative improvement algorithm shows smoother performance profiles which allow more refined control.
Lookahead and Pathology
 in Decision Tree Induction’’, IJCAI95
, 1995
"... The standard approach t decision tree in duction is a topdown greedy agonthm that makes locall} optimal irrevocable decisions at each node of a tree In this paper we empir•call} study an alternative approach in which the algorithms use onelevel loo kalie to deride what test to use at a node weyst ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
The standard approach t decision tree in duction is a topdown greedy agonthm that makes locall} optimal irrevocable decisions at each node of a tree In this paper we empir•call} study an alternative approach in which the algorithms use onelevel loo kalie to deride what test to use at a node weystematically compare using a very large number of rfal and artificial data sets the quality of dmsion trees induced by the greedv approach to that of trees induced using lookahead The main observations from our experments are (1) the greedv approach consistently produced trees that were just as at curate as trees produced with the much more expensive lookahead step and (n) we observed manv instances of pathology, le, lookalnad producrd trees that were both larger and less accurate than trees produced without it 1
SCALABLE CLASSIFICATION AND REGRESSION TREE CONSTRUCTION
, 2003
"... Automating the learning process is one of the long standing goals of Artificial Intelligence and its more recent specialization, Machine Learning. Supervised learning is a particular learning task in which the goal is to establish the connection between some of the attributes of the data made avai ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
Automating the learning process is one of the long standing goals of Artificial Intelligence and its more recent specialization, Machine Learning. Supervised learning is a particular learning task in which the goal is to establish the connection between some of the attributes of the data made available for learning, called attribute variables, and the remaining attributes called predicted attributes. This thesis is concerned exclusively with supervised learning using tree structured models: classification trees for predicting discrete outputs and regression trees for predicting continuous outputs. In the case of classification and regression trees most methods for selecting the split variable have a strong preference for variables with large domains. Our first contribution is a theoretical characterization of this preference and a general corrective method that can be applied to any split selection method. We further show how the general corrective method can be applied to the Gini gain for discrete variables when building kary splits. In the presence of large amounts of data, efficiency of the learning algorithms