## Multisensory oddity detection as bayesian inference (2009)

### Cached

### Download Links

Venue: | PLoS ONE |

Citations: | 4 - 1 self |

### BibTeX

@ARTICLE{Hospedales09multisensoryoddity,

author = {Timothy Hospedales and Sethu Vijayakumar},

title = {Multisensory oddity detection as bayesian inference},

journal = {PLoS ONE},

year = {2009}

}

### OpenURL

### Abstract

A key goal for the perceptual system is to optimally combine information from all the senses that may be available in order to develop the most accurate and unified picture possible of the outside world. The contemporary theoretical framework of ideal observer maximum likelihood integration (MLI) has been highly successful in modelling how the human brain combines information from a variety of different sensory modalities. However, in various recent experiments involving multisensory stimuli of uncertain correspondence, MLI breaks down as a successful model of sensory combination. Within the paradigm of direct stimulus estimation, perceptual models which use Bayesian inference to resolve correspondence have recently been shown to generalize successfully to these cases where MLI fails. This approach has been known variously as model inference, causal inference or structure inference. In this paper, we examine causal uncertainty in another important class of multi-sensory perception paradigm – that of oddity detection and demonstrate how a Bayesian ideal observer also treats oddity detection as a structure inference problem. We validate this approach by showing that it provides an intuitive and quantitative explanation of an important pair of multi-sensory oddity detection experiments – involving cues across and within modalities – for which MLI previously failed dramatically, allowing a novel unifying treatment of within and cross modal multisensory perception. Our successful application of structure inference models to the new ‘oddity detection ’ paradigm, and the resultant unified explanation of across and within modality cases provide

### Citations

1302 |
Information Theory, Inference and Learning Algorithms", University of Cambridge Press
- MacKay
- 2008
(Show Context)
Citation Context ...longer simply an estimation of a combined stimulus ^yh,v. Such an estimation is involved in solving the task, but ultimately the task effectively asks subjects to make a probabilistic model selection =-=[20,21]-=- between three models. (Note that this problem can also be understood as finding the most likely assignment of points in a clustering task. Specifically, consider mixture of Gaussian clustering of thr... |

582 | Bayesian interpolation
- MacKay
- 1991
(Show Context)
Citation Context ...longer simply an estimation of a combined stimulus ^yh,v. Such an estimation is involved in solving the task, but ultimately the task effectively asks subjects to make a probabilistic model selection =-=[20,21]-=- between three models. (Note that this problem can also be understood as finding the most likely assignment of points in a clustering task. Specifically, consider mixture of Gaussian clustering of thr... |

153 |
Perception as Bayesian inference
- Knill, Richards
- 1996
(Show Context)
Citation Context ...n the simpler task of stimulus estimation by cue combination. Our approach is parsimonious in that, within the research theme of investigating the extent to which human perception is Bayesian optimal =-=[23,24]-=-, models should use the same generative process as the perceptual experiment. By modelling the three sets of stimuli, including the selection of a probe stimulus and potential disassociation within th... |

112 | The ventriloquist effect results from near-optimal bimodal integration
- Alais, Burr
- 2004
(Show Context)
Citation Context ...eric band. The strongest conclusion that can be drawn is therefore that intra-modal perception shows a stronger tendency toward fusion than inter-modal perception. None of the basic theories proposed =-=(1,2,3,4)-=- explain the qualitative shape of the data well - good performance in the cues concordant quadrants 1&2 as well as a limited region of poor performance in the cues discordant quadrants 2&4. In particu... |

94 |
Knowledge representation and inference in similarity networks and Bayesian multinets
- Geiger, Heckerman
(Show Context)
Citation Context ...he theory and practice for modelling uncertain causal structure in inference tasks has a more extensive history in other fields. In artificial intelligence, the theory goes back to Bayesian multinets =-=[26]-=-, and is applied today, for example, in building artificial intelligence systems to explicitly understand correlations in multi-party conversations [11]. In radar tracking, this problem is known as da... |

86 |
Tracking in a cluttered environment with probabilistic data association
- Bar-Shalom, Tse
- 1975
(Show Context)
Citation Context ...d today, for example, in building artificial intelligence systems to explicitly understand correlations in multi-party conversations [11]. In radar tracking, this problem is known as data association =-=[27]-=-. Its solutions are used to sort out multiple radar detections, with uncertain causal relation to multiple aeroplanes, into a consistent and accurate estimate of the aircraft locations. A variety of r... |

45 |
MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415: 429–433
- MO, Banks
(Show Context)
Citation Context ...ization of it’s bark – because the optimal visual weight in this case is much larger. This has proven a good qualitative explanation of numerous experiments including audiovisual [3,4], visual-haptic =-=[5,6]-=-, texture-stereo [7,8] and texturemotion [9] pairs among others. The near optimal sensor fusion observed widely across these different pairs of senses suggests that this may be a common principle of s... |

26 | Causal inference in multisensory perception
- Körding, Beierholm, et al.
(Show Context)
Citation Context ...un to apply a complete Bayesian structure inference perspective [10,11] to experiments with such uncertainty [12,13,14], and have provided a good explanation for the perceptual process in these cases =-=[15,16]-=-. However, to date, all existing work on models of structure inference in human perception has been applied to paradigms involving direct estimation PLoS ONE | www.plosone.org 1 January 2009 | Volume ... |

25 |
Pouget A (2006) Bayesian inference with probabilistic population codes. Nat Neurosci 9
- WJ, JM, et al.
(Show Context)
Citation Context ...ute products of probability distributions; indeed, population codes represent-able by neurons with Poisson firing statistics would be particularly well suited for rapid computation of such operations =-=[36]-=-. Further experimental work is needed to confirm whether any of these proposed population coding models are actually implemented by biological neural networks. Conclusions In this paper, we have deriv... |

22 |
Mamassian P, Yuille A (2004) Object perception as Bayesian inference. Annu Rev Psychol 55:271--304
- Kersten
(Show Context)
Citation Context ...have declared that no competing interests exist. * E-mail: t.hospedales@ed.ac.uk Introduction Bayesian ideal observer modelling is an elegant and successful approach to understanding human perception =-=[1]-=-. One particular domain in which it has seen much success recently is that of understanding multisensory integration in human perception [2]. In this context, the ideal observer essentially specifies ... |

19 |
Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res 35:389–412
- MS, LT, et al.
- 1995
(Show Context)
Citation Context ...t and successful approach to understanding human perception [1]. One particular domain in which it has seen much success recently is that of understanding multisensory integration in human perception =-=[2]-=-. In this context, the ideal observer essentially specifies how the information from each sense should be optimally weighted in creating the unified percept of a particular source observed with multip... |

19 |
Creelman CD. 2005. Detection theory: a user’s guide
- NA
(Show Context)
Citation Context ...thin the domain of small discrepancies where mandatory fusion applies. Returning to the 3-alternative oddity task, a simple maximum likelihood estimator for uni-modal oddity is the ‘‘triangle rule’’ (=-=[25]-=-). This measures the distances between all three point combinations, discards the two points with minimum distance between them, and nominates the third point as odd. However, this does not provide an... |

17 |
Optimal integration of texture and motion cues to depth. Vision Res 39:3621–3629
- RA
- 1999
(Show Context)
Citation Context ...ual weight in this case is much larger. This has proven a good qualitative explanation of numerous experiments including audiovisual [3,4], visual-haptic [5,6], texture-stereo [7,8] and texturemotion =-=[9]-=- pairs among others. The near optimal sensor fusion observed widely across these different pairs of senses suggests that this may be a common principle of sensory integration in humans. However, these... |

16 |
Bulthoff HH. 2004. Merging the senses into a robust percept. Trends Cogn Sci. 8:162--169
- MO
(Show Context)
Citation Context ...n the simpler task of stimulus estimation by cue combination. Our approach is parsimonious in that, within the research theme of investigating the extent to which human perception is Bayesian optimal =-=[23,24]-=-, models should use the same generative process as the perceptual experiment. By modelling the three sets of stimuli, including the selection of a probe stimulus and potential disassociation within th... |

16 |
Pouget A (2004) The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci 27
- DC
(Show Context)
Citation Context ...aper to solve the oddity detection problem? Work on probabilistic population coding describes how neural populations could represent and compute with probability distributions such as those used here =-=[34,35]-=-. For the computations involved in multisensory integration, we need to compute products of probability distributions; indeed, population codes represent-able by neurons with Poisson firing statistics... |

12 |
Inference and computation with population codes. Annual Review of Neuroscience 26
- Pouget, Dayan, et al.
- 2003
(Show Context)
Citation Context ...aper to solve the oddity detection problem? Work on probabilistic population coding describes how neural populations could represent and compute with probability distributions such as those used here =-=[34,35]-=-. For the computations involved in multisensory integration, we need to compute products of probability distributions; indeed, population codes represent-able by neurons with Poisson firing statistics... |

8 |
Kamitani Y, Shimojo S (2000): Illusions. What you see is what you hear. Nature 408:788
- Shams
(Show Context)
Citation Context ...learly the most probable and detection is unreliable. Structure Inference All of the models discussed so far (Figs. 1 and 5) have assumed a fixed structure. Recent multisensory perception experiments =-=[12,13,14,17,22]-=-, have, however, presented subjects with what is essentially a variable causal structure with respect to the observation correspondence. It is therefore unsurprising that the simple fixed structure id... |

7 |
Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization
- PW, RA
(Show Context)
Citation Context ...f the basic theories (1, 2, and 4) which allow detection based on individual cues (magenta points are outside of the red lines in Fig. 4(b), quadrants 2&4). In both experiments, the last basic theory =-=(4)-=- of sequential combined and single cue detection also fails, as performance is worse than it predicts (magenta points outside the inner bounding box of lines in Fig. 4, quadrants 2&4). Since the poor ... |

7 |
JA (2003) Do humans optimally integrate stereo and texture information to slant? Vision Res 43:2539–2558
- DC, Saunders
(Show Context)
Citation Context ... because the optimal visual weight in this case is much larger. This has proven a good qualitative explanation of numerous experiments including audiovisual [3,4], visual-haptic [5,6], texture-stereo =-=[7,8]-=- and texturemotion [9] pairs among others. The near optimal sensor fusion observed widely across these different pairs of senses suggests that this may be a common principle of sensory integration in ... |

7 |
Beierholm U (2005) Sound-induced flash illusion as an optimal percept. Neuroreport 16
- Shams
- 1923
(Show Context)
Citation Context ...he experimental manipulation. This structure inference approach [11] has recently been used to understand other similarly perplexing experimental results in human audio-visual multisensory perception =-=[12,13,14,15,22]-=-. In summary, the standard maximum likelihood integration approach to sensor fusion has dramatically failed to explain the experimental data in [17]. This data can now be understood as result of the p... |

7 | Learning to integrate arbitrary signals from vision and touch - MO - 2007 |

6 | DM (2004) The loss function of sensorimotor learning - KP, Wolpert |

5 |
2003 Viewing geometry determines how vision and haptics combine in size perception
- Gepshtein, Banks
- 2002
(Show Context)
Citation Context ...lly, despite any apparent complexity, the new model introduces only one new free parameter. Further studies have investigated stereo-texture fusion [7,8] for slant perception and visual-haptic fusion =-=[6]-=- for size perception in greater detail, using simpler 2-alternative forced choice paradigms. These have provided further support for the near Bayesian optimality of human multisensory fusion, but only... |

5 |
Landy MS, Banks MS (2004) Slant from texture and disparity cues: optimal cue combination
- JM, SJ
(Show Context)
Citation Context ... because the optimal visual weight in this case is much larger. This has proven a good qualitative explanation of numerous experiments including audiovisual [3,4], visual-haptic [5,6], texture-stereo =-=[7,8]-=- and texturemotion [9] pairs among others. The near optimal sensor fusion observed widely across these different pairs of senses suggests that this may be a common principle of sensory integration in ... |

5 |
Banks MS, Landy MS (2002) Combining sensory information: mandatory fusion within, but not between, senses. Science 298:1627–1630
- JM, MO
(Show Context)
Citation Context ...gly unique paradigm that require careful considerations during modeling, how structure inference of causal uncertainty applies in this context, and how it can explain and unify a pair of experiments (=-=[17]-=-) where MLI previously failed dramatically. In the remainder of this section, we review standard MLI ideal observer modelling for sensor fusion, and show – by way of theoretical argument as well as a ... |

5 | Ernst MO (2006) Vision and touch are automatically integrated for the perception of sequences of events - Bresciani, Dammeier |

4 |
Unifying multisensory signals across time and space. Exp Brain Res 158
- MT, GE, et al.
- 2004
(Show Context)
Citation Context ... also infer the causal structure of the multisensory observations. Very recent work has begun to apply a complete Bayesian structure inference perspective [10,11] to experiments with such uncertainty =-=[12,13,14]-=-, and have provided a good explanation for the perceptual process in these cases [15,16]. However, to date, all existing work on models of structure inference in human perception has been applied to p... |

4 |
Toyoizumi T, Aihara K (2007) Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Neural computation 19: 3335–55
- Sato
(Show Context)
Citation Context ... coding models are actually implemented by biological neural networks. Conclusions In this paper, we have derived a Bayesian model for multisensory oddity detection which exploits structure inference =-=[11,15,16]-=-. With this model, we are able to understand the results of experiments on human multisensory oddity detection [17] which the classical maximum likelihood integration theory, and other simpler theorie... |

4 |
PV (2006) Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration. Proc Biol Sci 273
- NW, Heron, et al.
(Show Context)
Citation Context ...r, is insufficient, as it cannot explain complete segregation (complete non-interaction of the observations) observed in many experiments since the jointly Gaussian prior precludes this. Alternately, =-=[28]-=- proposes a joint prior with the special form of a Gaussian-uniform sum to reflect the fact that the observations in the environment are frequently very correlated but sometimes completely unrelated. ... |

4 |
Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant
- DC
- 2007
(Show Context)
Citation Context ..., see [11]). A related issue in theoretical modelling of perception is those scenarios in which we expect the prior distribution over an individual stimulus source to be a mixture. For example, Knill =-=[32]-=- considers the case of apparent visual ellipses which may have come from the set of true ellipses or the set of slanted circles. Combined with stereo cues for slant, estimation of ellipse slant also i... |

4 |
Banks MS (2005) The combination of vision and touch depends on spatial proximity
- Gepshtein, Burge, et al.
(Show Context)
Citation Context ...hey could also affect the parameters of the structure inference procedure. As an example, the strength of the fusion prior pc might decrease with the spatial discrepancy of the visual and haptic cues =-=[33]-=-. How might the perceptual system’s neural architecture perform the computations proposed in this paper to solve the oddity detection problem? Work on probabilistic population coding describes how neu... |

3 |
H (2001) Ideal cue combination for localizing texturedefined edges. J Opt Soc Am A Opt Image Sci Vis 18:2307–2320
- MS, Kojima
(Show Context)
Citation Context ... distribution. Formalizing Optimal Oddity Detection Ideal observer theories of cue combination in human multisensory perception have been tested extensively in the form of simple sensor fusion models =-=[2,5,6,9,19]-=-. Since these experiments are describable by a simple factored Gaussian parametric form (Fig. 1), the optimal computations to use for inference were those described by eqs. (1) and (2). However, the p... |

2 |
Visual localization ability influences cross-modal bias
- WD, MT, et al.
- 2003
(Show Context)
Citation Context ...resent all the regimes of the experiment. Moreover, the model would then not explicitly represent the structure C, which subjects do infer explicitly as reported in [17] and other related experiments =-=[12,13]-=-. Another reason for the perceptual system to explicitly represent and infer causal structure is that it may be of intrinsic interest. For example, in an audio-visual context, explicit knowledge of st... |

1 |
Vijayakumar S (2007) Structure inference for Bayesian multisensory perception and tracking
- Hospedales, Cartwright
(Show Context)
Citation Context ... inference (or causal inference) models which also infer the causal structure of the multisensory observations. Very recent work has begun to apply a complete Bayesian structure inference perspective =-=[10,11]-=- to experiments with such uncertainty [12,13,14], and have provided a good explanation for the perceptual process in these cases [15,16]. However, to date, all existing work on models of structure inf... |

1 |
Vijayakumar S (2008) Structure inference for Bayesian multisensory scene understanding
- Hospedales
(Show Context)
Citation Context ... inference (or causal inference) models which also infer the causal structure of the multisensory observations. Very recent work has begun to apply a complete Bayesian structure inference perspective =-=[10,11]-=- to experiments with such uncertainty [12,13,14], and have provided a good explanation for the perceptual process in these cases [15,16]. However, to date, all existing work on models of structure inf... |

1 | Perception of the human body from the inside out, chapter A Bayesian view on multimodal cue integration 105–131 - Ernst - 2005 |