Backpropagation is one of the most famous training algorithms for multilayer perceptrons. Unfortunately it can be very slow for practical applications. Over the last years many improvement strategies have been developed to speed up backpropagation. It's very difficult to compare these different techniques, because most of them have been tested on various specific data sets. Most of the reported results are based on some kind of tiny and artificial training sets like XOR, encoder or decoder. It's very doubtful if these results hold for more complicate practical application. In this report an overview of many different speedup techniques is given. All of them were assessed by a very hard practical classification task, which consists of a big medical data set. As you will see many of these optimized algorithms fail in learning the data set. 1 Introduction This report is intended to summarize our experience using many different speedup techniques for the backpropagation algorithm. We have...

Application and development of specialized machine learning techniques is gaining increasing attention in the intrusion detection community. A variety of learning techniques proposed for different intrusion detection problems can be roughly classified into two broad categories: supervised (classification) and unsupervised (anomaly detection and clustering). In this contribution we develop an experimental framework for comparative analysis of both kinds of learning techniques. In our framework we cast unsupervised techniques into a special case of classification, for which training and model selection can be performed by means of ROC analysis. We then investigate both kinds of learning techniques with respect to their detection accuracy and ability to detect unknown attacks.

This thesis investigates feedforward neural networks in the context of classification tasks with respect to the detection of patterns that do not belong to the same categories of patterns used to train the network. This refers to the problem of the detection and/or rejection of spurious or novel patterns. In particular, the multilayer perceptron network (MLP) trained with the backpropagation algorithm is examined in this respect and different strategies for improving its performance in the detection of spurious patterns are considered. The problem is investigated from different points of view that vary from the modification of the multilayer perceptron network with different configurations that make it more intrinsically able to detect spurious information, to the introduction of novel auxiliary mechanisms which, when integrated with the MLP network, can provide an overall enhancement in the system's rejection capabilities. These different network configurations are examined with respe...

Abstract — This paper presents an integrative approach to solve the coupled problem of reaching and grasping an object in a cluttered environment with a humanoid robot. While finding an optimal grasp is often treated independently from reaching to the object, in most situations it depends on how the robot can reach a pregrasp pose while avoiding obstacles. We tackle this problem by introducing the concept of task maps which represent the manifold of feasible grasps for an object. Rather than defining a single end-effector goal position, a task map defines a goal hyper volume in the task space. We show how to efficiently learn such maps using the Rapidly exploring Random Tree algorithm. Further, we generalise a previously developed motion optimisation scheme, based on a sequential attractor representation of motion, to cope with such task maps. The optimisation procedure incorporates the robot’s redundant whole body controller and uses analytic gradients to jointly optimise the motion costs (including criteria such as collision and joint limit avoidance, energy efficiency, etc.) and the choice of the grasp on the manifold of valid grasps. This leads to a preference of grasps which are easy to reach. The approach is demonstrated in two reach-grasp simulation scenarios with the humanoid robot ASIMO. I.

Xfuzzy is a CAD tool that eases the development of fuzzy systems from their conception to their final implementation. It is composed of a set of modules and programs that share a common specification language and cover the different stages of the design process. Modules for describing, verifying and tuning the behavior of the system are integrated within the environment. In addition to these features, common to other fuzzy design tools, a relevant characteristic of Xfuzzy is that it includes several synthesis facilities for implementing the system on either software or hardware. 1. Introduction The success of fuzzy logic applications in fields such as decision-making systems, control, image recognition and non-linear systems modelling has motivated the introduction of many fuzzy system development tools. However, most of these tools are tightly associated to specific architectures, producing implementations for predetermined microprocessors or microcontrollers, and offering a limited ...

Abstract — Efficient and reliable available bandwidth measurement remains an important goal for many applications. In this paper we introduce an empirical bandwidth estimation tool based on neural networks. Training the neural network on simulation data, it provides reliable estimation of physical and available bandwidth for simulated single and multi-hop networks, in laboratory environment and among the real world conditions of the ETOMIC testbed. I.

A development environment for fuzzy logic based inference systems is presented in this paper. CAD tools included within the environment simplify the tasks of specification, verification and synthesis of fuzzy systems. System specification is carried out with the help of a high-level description language. This specification is the input for all the verification and synthesis facilities provided by the environment. Verification tools allow to simulate the system behavior and to adjust the parameters that define the knowledge base. Finally, synthesis tools provide software and hardware implementations of fuzzy systems. Keywords: Fuzzy Systems, CAD Tools, Fuzzy Hardware. 1. INTRODUCTION Fuzzy set theory was introduced by L. A. Zadeh at the end of the 60's as a tool for describing the approximate inference mechanisms of human brain [1]. The knowledge base of a fuzzy system is stored in a symbolic way (like in expert systems) but it is processed numerically (like in artificial neural netw...

Training convolutional neural networks (CNNs) on large sets of high-resolution images is too computationally intense to be performed on commodity CPUs. Such architectures however achieve state-of-the-art results on low-resolution machine vision tasks such as the recognition of handwritten characters. We have adapted the inherent multi-level parallelism of CNNs for Nvidia’s CUDA GPU architecture to accelerate the training by two orders of magnitude. This dramatic speedup permits to apply CNN architectures to pattern recognition tasks on datasets with high-resolution natural images. 1

Abstract — This paper describes a new method for extracting symbolic rules from multilayer feedforward neural networks. Our approach is to encourage backpropagation to learn a sparser representation at the hidden layer and to use the improved representation to extract fewer, easier to understand rules. A new error term defined over the hidden layer is added to the standard sum of squared error so that the total squared distance between hidden activation vectors is increased. We show that this method helps extract fewer rules without decreasing classification accuracy in four publicly available data sets. I.

Recently we have developed a non-linear feature-domain noise reduction algorithm based on the minimum mean square error (MMSE) criterion on Mel-frequency cepstra (MFCC) for environment-robust speech recognition. Our novel algorithm operates on the power spectral magnitude of the filter-bank’s outputs and outperforms the log-MMSE spectral amplitude noise suppressor proposed by Ephraim and Malah in both recognition accuracy and efficiency as demonstrated on the Aurora-3 corpora. This paper serves two purposes. First, we show that the algorithm is effective on large vocabulary tasks with tri-phone acoustic models. Second, we report improvements on the suppression rule of the original MFCC-MMSE noise suppressor by smoothing the gain over the previous frames to prevent the abrupt change of the gain over frames and adjusting gain function based on the noise power so that the suppression is aggressive when the noise level is high and conservative when the noise level is low. We also propose an efficient and effective parameter tuning algorithm named step-adaptive discriminative learning algorithm (SADLA) to adjust the parameters used by the noise tracker and the suppressor. We observed a 46 % relative word error (WER) reduction on an in-house large-vocabulary noisy speech database with a clean trained model, which translates into a 16 % relative WER reduction over the original MFCC-MMSE noise suppressor, and 6 % relative WER reduction on the Aurora-3 corpora over our original MFCC-MMSE algorithm or 30 % relative WER reduction over the CMN baseline.