DMCA
ABSTRACT Optimal Multimodal Fusion for Multimedia Data Analysis
BibTeX
@MISC{Wu_abstractoptimal,
author = {Yi Wu and Kevin Chen-chuan Chang},
title = {ABSTRACT Optimal Multimodal Fusion for Multimedia Data Analysis},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Considerable research has been devoted to utilizing multimodal features for better understanding multimedia data. However, two core research issues have not yet been adequately addressed. First, given a set of features extracted from multiple media sources (e.g., extracted from the visual, audio, and caption track of videos), how do we determine the best modalities? Second, once a set of modalities has been identified, how do we best fuse them to map to semantics? In this paper, we propose a two-step approach. The first step finds statistically independent modalities from raw features. In the second step, we use super-kernel fusion to determine the optimal combination of individual modalities. We carefully analyze the tradeoffs between three design factors that affect fusion performance: modality independence, curse of dimensionality, and fusion-model complexity. Through analytical and empirical studies, we demonstrate that our two-step approach, which achieves a careful balance of the three design factors, can improve class-prediction accuracy over traditional techniques.
Keyphrases
multimedia data analysis abstract optimal multimodal fusion two-step approach design factor multiple medium source optimal combination individual modality multimodal feature caption track traditional technique super-kernel fusion empirical study core research issue careful balance second step understanding multimedia data independent modality class-prediction accuracy considerable research modality independence fusion-model complexity raw feature first step fusion performance
