## A System for Induction of Oblique Decision Trees (1994)

### Cached

### Download Links

- [www.cs.jhu.edu]
- [www.jair.org]
- [www.cs.cmu.edu]
- DBLP

### Other Repositories/Bibliography

Venue: | Journal of Artificial Intelligence Research |

Citations: | 250 - 13 self |

### BibTeX

@ARTICLE{Murthy94asystem,

author = {Sreerama K. Murthy and Simon Kasif and Steven Salzberg},

title = {A System for Induction of Oblique Decision Trees},

journal = {Journal of Artificial Intelligence Research},

year = {1994},

volume = {2},

pages = {1--32}

}

### Years of Citing Articles

### OpenURL

### Abstract

This article describes a new system for induction of oblique decision trees. This system, OC1, combines deterministic hill-climbing with two forms of randomization to find a good oblique split (in the form of a hyperplane) at each node of a decision tree. Oblique decision tree methods are tuned especially for domains in which the attributes are numeric, although they can be adapted to symbolic or mixed symbolic/numeric attributes. We present extensive empirical studies, using both real and artificial data, that analyze OC1's ability to construct oblique trees that are smaller and more accurate than their axis-parallel counterparts. We also examine the benefits of randomization for the construction of oblique decision trees. 1. Introduction Current data collection technology provides a unique challenge and opportunity for automated machine learning techniques. The advent of major scientific projects such as the Human Genome Project, the Hubble Space Telescope, and the human brain mappi...

### Citations

4957 |
C4.5: programs for machine learning
- Quinlan
- 1992
(Show Context)
Citation Context ...ble ways. Figure 3 illustrates these upper limits for two points in two dimensions. For axis-parallel splits, there are only n \Delta d distinct possibilities, and axis-parallel methods such as C4.5 (=-=Quinlan, 1993-=-a) and CART (Breiman et al., 1984) can exhaustively search for the best split at each node. The problem of searching for the best oblique split is therefore much more difficult than that of searching ... |

3928 |
Classification and Regression Trees
- Breiman, Friedman, et al.
- 1984
(Show Context)
Citation Context ...ssarily restrict oblique decision trees to numeric domains. Several researchers have studied the problem of converting symbolic (unordered) domains to numeric (ordered) domains and vice versa; e.g., (=-=Breiman et al., 1984-=-; Hampson & Volper, 1986; Utgoff & Brodley, 1990; Van de Merckt, 1992, 1993). To keep the discussion simple, however, we will assume that all attributes have numeric values. Induction of Oblique Decis... |

3559 | Optimization by simulated annealing - Kirkpatrick, Gelatt, et al. - 1983 |

3364 | Induction on decision trees
- Quinlan
- 1986
(Show Context)
Citation Context ...thms have been introduced in the last decade. Much of this work has concentrated on decision trees in which each node checks the value of a single attribute (Breiman, Friedman, Olshen, & Stone, 1984; =-=Quinlan, 1986-=-, 1993a). Quinlan initially proposed decision trees for classification in domains with symbolic-valued attributes (1986), and later extended them to numeric domains (1987). When the attributes are num... |

741 | Aha, “UCI repository of machine learning data bases,” http: //www.ics.uci.edu/~mlearn/MLRepository.html - Murphy, W - 1992 |

438 | Very simple classification rules perform well on most commonly used datasets
- Holte
- 1993
(Show Context)
Citation Context ..., and therefore the advantages of randomization may not be detectable. It is known that many of the commonly-used data sets from the UCI repository are easy to learn with very simple representations (=-=Holte, 1993); therefo-=-re those data sets may not be ideal for our purposes. Thus we created a number of artificial data sets that present different problems for learning, and for which we know the "correct" conce... |

357 | A practical approach to feature selection - Kira, Rendell - 1992 |

308 |
Learning Efficient Classification Procedures and their ,ipplication to Chess Endgame
- Quinlan
(Show Context)
Citation Context ...classification and other tasks since the 1960s (Moret, 1982; Safavin & Landgrebe, 1991). In the 1980's, Breiman et al.'s book on classification and regression trees (CART) and Quinlan 's work on ID3 (=-=Quinlan, 1983-=-, 1986) provided the foundations for what has become a large body of research on one of the central techniques of experimental machine learning. Many variants of decision tree (DT) algorithms have bee... |

238 |
Regression Diagnostics: Identifying Influential Data and Sources of Collinearity
- Belsley, Kuh, et al.
- 1980
(Show Context)
Citation Context .... The category variable (median value of owner-occupied homes) is actually continuous, but we discretized it so that category = 1 if value ! $21000, and 2 otherwise. For other uses of this data, see (=-=Belsley, 1980-=-; Quinlan, 1993b). Diabetes diagnosis. This data catalogs the presence or absence of diabetes among Pima Indian females, 21 years or older, as a function of eight numeric-valued attributes. The origin... |

210 | Robust linear programming discrimination of two linearly inseparable sets - Bennett, Mangasarian - 1992 |

203 | Training a 3-node neural network is NPcomplete - Blum, Rivest - 1992 |

202 |
Boolean feature discovery in empirical learning
- Pagallo, Haussler
- 1990
(Show Context)
Citation Context ... The right side shows the partitioning that this tree creates in the attribute space. Researchers have also studied decision trees in which the test at a node uses boolean combinations of attributes (=-=Pagallo, 1990-=-; Pagallo & Haussler, 1990; Sahami, 1993) and linear combinations of attributes (see Section 2). Different methods for measuring the goodness of decision tree nodes, as well as techniques for pruning ... |

195 |
Classi - cation and regression trees
- Breiman, Friedman, et al.
- 1984
(Show Context)
Citation Context ... Many variants of decision tree (DT) algorithms have been introduced in the last decade. Much of this work has concentrated on decision trees in which each nodechecks the value of a single attribute (=-=Breiman, Friedman, Olshen, & Stone, 1984-=-; Quinlan, 1986, 1993a). Quinlan initially proposed decision trees for classi cation in domains with symbolic-valued attributes (1986), and later extended them to numeric domains (1987). When the attr... |

185 |
Constructing optimal binary decision trees is NPcomplete
- Hyafil, Rivest
- 1976
(Show Context)
Citation Context ...e optimal test (with respect to an impurity measure) for each node of a tree, the complete tree may not be optimal: as is well known, the problem of finding the smallest tree is NP-Complete (Hyafil & =-=Rivest, 1976). Thus ev-=-en axis-parallel decision tree methods do not produce "ideal" decision trees. Quinlan has suggested that his windowing algorithm might be used as a way of introducing randomization into C4.5... |

171 | Second order derivatives for network pruning: Optimal brain surgeon - Hassibi, Stork - 1992 |

170 | A Nearest Hyperrectangle Learning Method - Salzberg - 1991 |

167 |
An Empirical Comparison of Selection Measures for Decision Tree Induction
- MINGERS
- 1989
(Show Context)
Citation Context ...impurity, the difference being that goodness values should be maximized while impurity should be minimized. Many different measures of impurity have been studied (Breiman et al., 1984; Quinlan, 1986; =-=Mingers, 1989-=-b; Buntine & Niblett, 1992; Fayyad & Irani, 1992; Heath et al., 1993b). The OC1 system is designed to work with a large class of impurity measures. Stated simply, if the impurity measure uses only the... |

166 |
An empirical comparison of pruning methods for decision tree induction
- Mingers
- 1989
(Show Context)
Citation Context ...impurity, the difference being that goodness values should be maximized while impurity should be minimized. Many different measures of impurity have been studied (Breiman et al., 1984; Quinlan, 1986; =-=Mingers, 1989-=-b; Buntine & Niblett, 1992; Fayyad & Irani, 1992; Heath et al., 1993b). The OC1 system is designed to work with a large class of impurity measures. Stated simply, if the impurity measure uses only the... |

151 | Learning Machines
- Nilsson
- 1965
(Show Context)
Citation Context ...em (Utgoff & Brodley, 1991; Brodley & Utgoff, 1992), which is a successor to the Perceptron Tree method (Utgoff, 1989; Utgoff & Brodley, 1990). Each internal node in an LMDT tree is a Linear Machine (=-=Nilsson, 1990-=-). The training algorithm presents examples repeatedly at each node until the linear machine converges. Because convergence cannot be guaranteed, LMDT uses heuristics to determine when the node has st... |

126 | An empirical comparison of pattern recognition, neural nets, and machine learning classification methods - Weiss - 1989 |

120 | Overfitting avoidance as bias
- Schaffer
- 1993
(Show Context)
Citation Context ...rees as well as other types of machine learning systems (Quinlan, 1987; Niblett, 1986; Cestnik, Kononenko, & Bratko, 1987; Kodratoff & Manago, 1987; Cohen, 1993; Hassibi & Stork, 1993; Wolpert, 1992; =-=Schaffer, 1993-=-). For the OC1 system we implemented an existing pruning method, but note that any tree pruning method will work fine within OC1. Based on the experimental evaluations of Mingers (1989a) and other wor... |

119 | Multivariate decision trees - Brodley, Utgoff - 1995 |

115 | Combining instance-based and model-based learning," presented at
- Quinlan
- 1993
(Show Context)
Citation Context ...ble ways. Figure 3 illustrates these upper limits for two points in two dimensions. For axis-parallel splits, there are only n \Delta d distinct possibilities, and axis-parallel methods such as C4.5 (=-=Quinlan, 1993-=-a) and CART (Breiman et al., 1984) can exhaustively search for the best split at each node. The problem of searching for the best oblique split is therefore much more difficult than that of searching ... |

97 |
A further comparison of splitting rules for decision-tree induction
- Buntine, Niblett
- 1992
(Show Context)
Citation Context ...urbations before forcing the perturbation algorithm to halt. We also included axis-parallel CART and C4.5 in our comparisons. We used the implementations of these algorithms from the IND 2.1 package (=-=Buntine, 1992). The def-=-ault cart0 and c4.5 "styles" defined in the package were used, without altering any parameter settings. The cart0 style uses the Twoing Rule and 0-SE cost complexity pruning with 10-fold cro... |

92 | Using Decision Trees to Improve Case-Based Learning
- Cardie
- 1993
(Show Context)
Citation Context ...ant attributes Irrelevant attributes pose a significant problem for most machine learning methods (Breiman et al., 1984; Aha, 1990; Almuallin & Dietterich, 1991; Kira & Rendell, 1992; Salzberg, 1992; =-=Cardie, 1993-=-; Schlimmer, 1993; Langley & Sage, 1993; Brodley & Utgoff, 1994). Decision tree algorithms, even axis-parallel ones, can be confused by too many irrelevant attributes. Because oblique decision trees l... |

91 | Hedonic prices and the demand for clean air - Harrison, Rubinfeld - 1978 |

85 | Using the ADAP learning algorithm to forecast the onset of diabetes mellitus - Smith, Everhart, et al. - 1988 |

83 | system for induction of oblique decision trees - Kasif, Salzberg - 1994 |

76 |
A study of instance-based algorithms for supervised learning tasks: mathematical, empirical, and psychological evaluations
- Aha
- 1990
(Show Context)
Citation Context ...Complexity pruning, see Breiman et al. (1984) or Mingers (1989a). 3.3.3 Irrelevant attributes Irrelevant attributes pose a significant problem for most machine learning methods (Breiman et al., 1984; =-=Aha, 1990-=-; Almuallin & Dietterich, 1991; Kira & Rendell, 1992; Salzberg, 1992; Cardie, 1993; Schlimmer, 1993; Langley & Sage, 1993; Brodley & Utgoff, 1994). Decision tree algorithms, even axis-parallel ones, c... |

69 | Pattern recognition via linear programming: Theory and application to medical diagnosis. Large-scale numerical optimization - Mangasarian, Setiono, et al. - 1990 |

67 |
Constructing decision trees in noisy domains
- Niblett
- 1986
(Show Context)
Citation Context ...g the data. Many studies have found that judicious pruning results in both smaller and more accurate classifiers, for decision trees as well as other types of machine learning systems (Quinlan, 1987; =-=Niblett, 1986-=-; Cestnik, Kononenko, & Bratko, 1987; Kodratoff & Manago, 1987; Cohen, 1993; Hassibi & Stork, 1993; Wolpert, 1992; Schaffer, 1993). For the OC1 system we implemented an existing pruning method, but no... |

64 | E ciently inducing determinations: A complete and systematic search algorithm that uses optimal pruning
- Schlimmer
- 1993
(Show Context)
Citation Context ... Irrelevant attributes pose a significant problem for most machine learning methods (Breiman et al., 1984; Aha, 1990; Almuallin & Dietterich, 1991; Kira & Rendell, 1992; Salzberg, 1992; Cardie, 1993; =-=Schlimmer, 1993-=-; Langley & Sage, 1993; Brodley & Utgoff, 1994). Decision tree algorithms, even axis-parallel ones, can be confused by too many irrelevant attributes. Because oblique decision trees learn the coeffici... |

63 |
Decision Trees and Diagrams
- Moret
- 1982
(Show Context)
Citation Context ...ssification by a sequence of simple, easy-to-understand tests whose semantics is intuitively clear to domain experts. Decision trees have been used for classification and other tasks since the 1960s (=-=Moret, 1982-=-; Safavin & Landgrebe, 1991). In the 1980's, Breiman et al.'s book on classification and regression trees (CART) and Quinlan 's work on ID3 (Quinlan, 1983, 1986) provided the foundations for what has ... |

58 |
Perceptron trees: a case study in hybrid concept representations
- Utgoff
- 1988
(Show Context)
Citation Context ...t uses a very different approach from CART-LC, is the Linear Machine Decision Trees (LMDT) system (Utgoff & Brodley, 1991; Brodley & Utgoff, 1992), which is a successor to the Perceptron Tree method (=-=Utgoff, 1989-=-; Utgoff & Brodley, 1990). Each internal node in an LMDT tree is a Linear Machine (Nilsson, 1990). The training algorithm presents examples repeatedly at each node until the linear machine converges. ... |

55 | OC1: A Randomized Induction of Oblique Decision Trees
- Murthy, Kasif, et al.
- 1993
(Show Context)
Citation Context ...perturbations will occur at each node. This constant can be set by the user. Pstag is reset to 1 every time the global impurity measure is improved. Murthy, Kasif & Salzberg Our previous experiments (=-=Murthy et al., 1993-=-) indicated that the order of perturbation of the coefficients does not affect the classification accuracy as much as other parameters, especially the randomization parameters (see below). Since none ... |

36 | Linear Machine decision trees - Utgoff, Brodley - 1991 |

35 | Linear function neurons: structure and training - Hampson, Volper - 1986 |

30 | Fast training algorithms for multilayer neural nets - Brent - 1991 |

30 | Multivariate versus univariate decision trees - Brodley, Utgoff - 1992 |

29 | An incremental method for finding multivariate splits for decision trees - Utgoff, Brodley - 1990 |

24 |
Adaptive Decision Tree Algorithms for Learning from Examples
- Pagallo
- 1990
(Show Context)
Citation Context ... The right side shows the partitioning that this tree creates in the attribute space. Researchers have also studied decision trees in which the test at a node uses boolean combinations of attributes (=-=Pagallo, 1990-=-; Pagallo & Haussler, 1990; Sahami, 1993) and linear combinations of attributes (see Section 2). Different methods for measuring the goodness of decision tree nodes, as well as techniques for pruning ... |

23 | Multicategory discrimination via linear programming. Optimization Methods and SoBware - Bennett, Mangasarian - 1994 |

23 |
Very simple classi cation rules perform well on most commonly used datasets
- Holte
- 1993
(Show Context)
Citation Context ..., and therefore the advantages of randomization may not be detectable. It is known that many of the commonly-used data sets from the UCI repository are easy to learn with very simple representations (=-=Holte, 1993-=-); therefore those data sets may not be ideal for our purposes. Thus we created a number of arti cial data sets that present di erent problems for learning, and for which we know the \correct" concept... |

22 | A machine learning method for generation of a neuralnetwork architecture: A continuous id3 algorithm - Cios - 1992 |

22 | On Randomization in Sequential and Distributed Algorithms - Gupta, Smolka, et al. - 1994 |

20 | A geometric framework for machine learning - Heath - 1992 |

18 |
Learning E cient Classi cation Procedures and their Application to Chess End Games
- Quinlan
- 1983
(Show Context)
Citation Context ...or classi cation and other tasks since the 1960s (Moret, 1982; Safavin & Landgrebe, 1991). In the 1980's, Breiman et al.'s book on classi cation and regression trees (CART) and Quinlan's work on ID3 (=-=Quinlan, 1983-=-, 1986) provided the foundations for what has become a large body of research on one of the central techniques of experimental machine learning. Many variants of decision tree (DT) algorithms have bee... |

16 |
nets and short paths: Optimising Neural Computation
- Frean
- 1990
(Show Context)
Citation Context ...guaranteed, LMDT uses heuristics to determine when the node has stabilized. To make the training stable even when the set of training instances is not linearly separable, a "thermal training"=-=; method (Frean, 1990-=-) is used, similar to simulated annealing. A third system that creates oblique trees is Simulated Annealing of Decision Trees (SADT) (Heath et al., 1993b) which, like OC1, uses randomization. SADT use... |

12 | A survey of decision tree classifier methodology - Safavin, Landgrebe - 1991 |

12 | Decision trees in numerical attribute spaces - Merckt |