this article. Neural Computation 13, 2049--2074 (2001) c 2001 Massachusetts Institute of Technology noise, and so on), which affect different cues in a different manner. A cue that is usually reliable for computing a particular piece of information may become useless or even harmful in certain situations and is better suppressed. Studies with humans suggest that suppression and recalibration of discordant sensory modalities do occur (Bower, 1974; Murphy, 1996); when there is disagreement among cues, the brain reacts by adapting its integration strategy. Importantly, no external teacher tells the brain which cues have to be suppressed or recalibrated. The adaptation works in a self-organized manner. This phenomenon has not been explained satisfactorily yet. In the engineering literature, little work has been done on self-organized adaptive sensory integration either, although the importance of adaptation has recently been stressed by several authors (Murphy, 1996; Dasarathy, 1997; Kam, Zhu, & Kalata, 1997)

A mechanism for the self-organized integration of different adaptive cues is proposed. In Democratic Integration the cues agree on a result and each cue adapts towards the result agreed upon. A technical formulation of this scheme is employed in a face tracking system. The self-organized adaptivity leads to suppression and recalibration of discordant cues. Experiments show that the system is robust to sudden changes in the environment as long as the changes disrupt only a minority of cues at the same time, although all cues may be affected in the long run. 1. Introduction The integration of information stemming from different cues or modalities is among the most fundamental problems of perception in biological and artificial systems. Due to frequent changes in complex environments, the integration of cues has to be adaptive. However, there usually is no teacher available to guide the adaptation. The agent has to figure out on his own which cues are reliable for the given task in the ...

Information fusion is a mandatory prerequisite for cognitive vision systems. These are vision systems that apply reasoning and learning on different levels of abstraction and correspondingly have to deal with hypotheses from different categorical domains. Following some principles of human cognition, we present an approach to information fusion that closely couples reasoning and representation. We will discuss how processes like probabilistic contextual reasoning as well as functional and nonfunctional requirements in storing data from different sources can be integrated by a unified XML based data representation. Due to the interaction between active processes and data storage, we call our approach an active memory. Performance results of an implemented system as well as an evaluation of data fusion from contextual inference will be presented.

We present a framework for merging the results of independent featurebased motion trackers using a classification based approach. We demonstrate the efficacy of the framework using corner trackers as an example. The major

This study investigated recognition systems that combine loosely coupled modalities, integrating eye movements in an Automatic Speech Recognition (ASR) system as an exemplar. A probabilistic framework for combining modalities was formalised and applied to the specific case of integrating eye movement and speech. A corpus of a matched eye movement and related spontaneous conversational British English speech for a visual-based, goal-driven task was collected. This corpus enabled the relationship between the modalities to be verified. Robust extraction of visual attention from eye movement data was investigated using Hidden Markov Models and Hidden Semi-Markov Models. Gaze-contingent ASR systems were developed from a research-grade baseline ASR system by redistributing language model probability mass according to the visual attention. The best performing systems maintained the Word Error Rates but showed an increase in the Figure of Merit- a measure of the keyword spotting accuracy and integration success. The core values of this work may be useful for developing robust multimodal decoding system functions.

The human brain has to integrate the inputs it receives from different sensory modalities into a coherent description of its environment. This integration is often adaptive, showing recalibration or suppression of discordant sensory modalities. This paper proposes a qualitative theory of sensory integration which relates these adaptation phenomena to the anatomy of the neocortex and a rapid reversible synaptic mechanism, as proposed in von der Malsburg's correlation theory of brain function [19].

: We focus on the detection of incipient faults in an internal combustion engine using a minimum number of sensory information. Inducing several faults in a 4 stroke diesel engine, cylinder pressure (P) and vibration (V) data are acquired. Two sets of artificial neural nets (ANN) are trained separately, using features from the pressure and vibration data. Both sets of nets show very good fault detection capabilities, thus demonstrating an alternative to the multi-sensory approach commonly adopted in fault diagnosis. In a separate study, P and V are fused together at the signal level and then used to train another set of ANNs which is shown to exhibit better reliability than either system. In the final study, the outputs of the 3 systems (P, V and fused P and V), are combined together in a majority voting system which outperforms all of its constituents in its diagnostic abilities, successfully identifying 2854 out of 3000 test cases with a confidence level of 90%. Keywords: cylinder p...

Both animals and robotic agents must combine information from multiple modalities to behave appropriately in complex environments. This study investigates the interaction of vision and olfaction in flying fruit flies (Drosophila melanogaster). Flies require vertical visual contrast to locate attractive odour sources (Frye et al, 2003). The reason for this is unknown; this project aims to employ a modelling approach to derive an algorithmic account of the mechanisms responsible for this observation. The fly's visuomotor control system is modelled, using Reichardt (1969) elementary motion detectors and analogues of the lobula plate tangential cells to detect optic flow patterns. Ways in which olfaction could modulate this behaviour will then be investigated. A particular hypothesis to be assessed is that the fly employs path integration based on visual odometry. A robotic model will be implemented to facilitate evaluation of behavioural algorithms in a physical odour plume, as this is difficult to accurately simulate. In order to provide data to constrain and validate the models, a system to record flies ' flight trajectories in 3D is also implemented. Finally, a combination of high-level neural modelling and experiments on mutant flies will be used to investigate the neural basis of sensor fusion in the fly. 1 1.

Existing techniques for sensor validation and sensor fusion are often based on analytical sensor models. Such models can be arbitrarily complex and consequently Gaussian distributions are often assumed, generally with a detrimental effect on overall system performance. A holistic approach has therefore been adopted in order to develop two novel and complementary approaches to sensor validation and fusion based on empirical data. The first uses the Nadaraya-Watson kernel estimator to provide competitive sensor fusion. The new algorithm is shown to reliably detect and compensate for bias errors, spike errors, hardover faults, drift faults and erratic operation, affecting up to three of the five sensors in the array. The inherent smoothing action of the kernel estimator provides effective noise cancellation and the fused result is more accurate than the single ‘best sensor’. A Genetic Algorithm has been used to optimise the Nadaraya-Watson fuser design. The second approach uses analytical redundancy to provide the on-line sensor status output µH∈[0,1], where µH=1 indicates the sensor output is valid and µH=0 when the sensor has failed. This fuzzy measure is derived from change detection parameters

An ARTMAP neural network is used to integrate visual information and ultrasonic sensory information on a B14 mobile robot. Training samples for the neural network are acquired without human intervention. Sensory snapshots are retrospectively associated with the distance to the wall, provided by on-board odometry as the robot travels in a straight line. The goal is to produce a more accurate measure of distance than is provided by the raw sensors. The neural network effectively combines sensory sources both within and between modalities. The improved distance percept is used to produce occupancy grid visualizations of the robot's environment. The maps produced point to specific problems of raw sensory information processing and demonstrate the benefits of using a neural network system for sensor fusion. Keywords: Mobile robot, neural network, sensor fusion, occupancy grid, sonar, range 1. INTRODUCTION: NEURAL SENSOR FUSION Mobile robots require accurate representations of their surro...