Visual speech recognition with loosely synchronized feature streams

DMCA

Visual speech recognition with loosely synchronized feature streams (2005)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Kate Saenko , Karen Livescu , Michael Siracusa , Kevin Wilson , James Glass , Trevor Darrell

Venue:	In Proc. ICCV
Citations:	25 - 6 self

BibTeX

@INPROCEEDINGS{Saenko05visualspeech,
    author = {Kate Saenko and Karen Livescu and Michael Siracusa and Kevin Wilson and James Glass and Trevor Darrell},
    title = {Visual speech recognition with loosely synchronized feature streams},
    booktitle = {In Proc. ICCV},
    year = {2005},
    pages = {1424--1431}
}

OpenURL

Abstract

We present an approach to detecting and recognizing spoken isolated phrases based solely on visual input. We adopt an architecture that first employs discriminative detection of visual speech and articulatory features, and then performs recognition using a model that accounts for the loose synchronization of the feature streams. Discriminative classifiers detect the subclass of lip appearance corresponding to the presence of speech, and further decompose it into features corresponding to the physical components of articulatory production. These components often evolve in a semi-independent fashion, and conventional visemebased approaches to recognition fail to capture the resulting co-articulation effects. We present a novel dynamic Bayesian network with a multi-stream structure and observations consisting of articulatory feature classifier scores, which can model varying degrees of co-articulation in a principled way. We evaluate our visual-only recognition system on a command utterance task. We show comparative results on lip detection and speech/nonspeech classification, as well as recognition performance against several baseline systems. 1.

Keyphrases

visual speech recognition synchronized feature stream recognition performance physical component lip appearance articulatory feature visual-only recognition system novel dynamic bayesian network principled way visual speech feature stream discriminative detection articulatory feature classifier score lip detection speech nonspeech classification several baseline system co-articulation effect discriminative classifier comparative result conventional visemebased approach multi-stream structure articulatory production command utterance task visual input loose synchronization semi-independent fashion

DMCA