Abstract

While modular architectures have desirable properties, integrating the outputs of many modules into a unified representation is not a trivial issue. In this paper we examine catastrophic fusion, a problem that occurs when modules are fused in incorrect context conditions. This problem has become especially apparent in the current research on automatic recognition of multimodal signals and thus has important practical as well as theoretical relevance. Catastrophic fusion arises because modules make implicit assumptions and thus operate correctly only within a certain context. Practice shows that when modules are tested in contexts inconsistent with their assumptions, their influence on the fused product tends to increase, with catastrophic results. We propose a principled solution to this problem based upon Bayesian ideas of competitive models. We study the approach analytically on a classic Gaussian discrimination task and then apply it to a realistic problem on audiovisual speech reco...

Citations