Abstract — The paired-difference t-test is commonly used in the machine learning community to determine whether one learning algorithm is better than another on a given learning task. This paper suggests the use of the permutation test instead because it calculates the exact p-value instead of an estimate. The permutation test is also distribution free and the time complexity is trivial for the commonly used 10-fold cross-validation paireddifference test. Results of experiments on real-world problems suggest it is not uncommon to see the t-test estimate deviate up to 30-50 % from the exact p-value. I.

Evaluation strategies to assess the effectiveness of legal knowledge based systems enable strengths and limitations of systems to be accurately articulated. This facilitates efforts in the research community to develop systems and also promotes the adoption of research prototypes in the commercial world. However, evaluation strategies for systems that operate in a domain as complex as law are difficult to specify. In this paper, we present an evaluation framework put forward by Reich and describe how this motivated the evaluation of our systems in Australian family law. Strategies surveyed include a comparison of linear regression with neural networks, user acceptance surveys, a comparison of system predictions with those from past cases, and a comparison of system outputs with those proposed by a panel of lawyers. Specific criteria for the evaluation of explanation facilities are also described.. Keywords Evaluation, legal knowledge based systems. 1. INTRODUCTION Reich [10] notes ...

Abstract—Segmentation of an image into regions and the labeling of the regions is a challenging problem. In this paper, an approach that is applicable to any set of multifeature images of the same location is derived. Our approach applies to, for example, medical images of a region of the body; repeated camera images of the same area; and satellite images of a region. The segmentation and labeling approach described here uses a set of training images and domain knowledge to produce an image segmentation system that can be used without change on images of the same region collected over time. How to obtain training images, integrate domain knowledge, and utilize learning to segment and label images of the same region taken under any condition for which a training image exists is detailed. It is shown that clustering in conjunction with image processing techniques utilizing an iterative approach can effectively identify objects of interest in images. The segmentation and labeling approach described here is applied to color camera images and two other image domains are used to illustrate the applicability of the approach. Index Terms—Classification, color image processing, expert system, knowledge extraction. I.

In computational analysis in scientific domains, images are often compared based on their features, e.g., size, depth and other domain-specific aspects. Certain features may be more significant than others while comparing the images and drawing corresponding inferences for specific applications. Though domain experts may have subjective notions of similarity for comparison, they seldom have a distance function that ranks the image features based on their relative importance. We propose a method called FeaturesRank for learning such a distance function in order to capture the semantics of the images. We are given training samples with pairs of images and the extent of similarity identified for each pair. Using a guessed initial distance function, FeaturesRank clusters the given images in levels. It then adjusts the distance function based on the error between the clusters and training samples using heuristics proposed in this paper. The distance function that gives the lowest error is the output. This contains the features ranked in the order most appropriate the domain. FeaturesRank is evaluated with real image data from nanotechnology and bioinformatics. The results of our evaluation are presented in the paper. 1.

: Neural networks (NN) have become to be general tools for modeling functional relationships in engineering. They are used to model the behavior of products and the properties of processes. Nevertheless, their use is often ad hoc. This paper provides a sound basis for using NN as tools for modeling functional relationships implicit in empirical engineering data. First, a clear definition of a modeling task is given, followed by reviewing the theoretical modeling capabilities of NN and NN model estimation. Subsequently, a procedure for using NN in engineering practice is described and illustrated with an example of modeling marine propeller behavior. Particular attention is devoted to better estimation of model quality, insight on the influence of measurements error on model quality, and the use of advanced methods such as stacked generalization and ensemble modeling to further improve model quality. Using a new method of ensemble of SG(k \Gamma NN ), one could improve the qu...

: In the course of data modeling, many models could be created. Much work has been done on formulating guidelines for model selection. However, by and large, these guidelines are conservative or too specific. Instead of using general guidelines, models could be selected for a particular task based on statistical tests. When selecting one model, others are discarded. Instead of losing potential sources of information, models could be combined to yield better performance. We review the basics of model selection and combination and discuss their differences. Two examples of opportunistic and principled combinations are presented. The first demonstrates that mediocre quality models could be combined to yield significantly better performance. The latter is the main contribution of the paper; it describes and illustrates a novel heuristic approach called the SG(k-NN) ensemble for the generation of good quality and diverse models that can even improve excellent quality models. Key...

In this companion paper, we formally introduce STRAT, a stratification centric methodology for the empirical evaluation of classification systems. The motivating criteria for STRAT's development are discussed, as well as the potential consequences of departing from some common statistical assumptions made when applying more traditional methods. STRAT uses an established replicate statistical technique called balanced repeated replication, or BRR, that does not require the i.i.d. assumption needed for bootstrapping, jackknifing, or binomial techniques.

Abstract: In most cases visual inspection of the hot strip by an inspector (in real time or videotaped) is a difficult task. The issues in this project study are data modeling, Machine Learning (ML) model- neural networks (NN) modeling and reliability of such models for automatic detection and classification of defects of hot strips. The proposed study intends to develop general guidelines for developing NN model for automatic surface inspection for hot strip mills. Introduction: In steel industry, visual inspection of the hot strip by an inspector is, in most cases not possible because of the high speeds and high temperature involved. In recent times, only video monitors and video recorders have been used where inspectors check on-line or taped video sequences for defects. In this way, only small parts of the strip’s top or bottom side are viewed. Additionally, this visual inspection is subjective and dependent on a large number of human

Inductive machine learning of optimal modular structures: Estimating solutions using support vector machines While structural optimisation is usually handled by iterative methods requiring repeated samples of a physics-based model, this process can be computationally demanding. Given a set of previously optimised structures of the same topology, this paper uses inductive learning to replace this optimisation process entirely by deriving a function that directly maps any given load to an optimal geometry. A support vector machine is trained to determine the optimal geometry of individual modules of a space frame structure given a specified load condition. Structures produced by learning are compared against those found by a standard gradient descent optimisation, both as individual modules and then as a composite structure. The primary motivation for this is speed, and results show the process is highly efficient for cases in which similar optimisations must be performed repeatedly. The function learned by the algorithm can approximate the result of optimisation very closely after sufficient training, and has also been found effective at generalising the underlying optima to produce structures that perform better than those found by standard iterative methods. Keywords Machine learning; support vector machines; structures; optimisation 1.0