Abstract

In this paper we show how recognition performance in automated speech perception can be significantly improved by additional Lipreading, so called "speech-reading ". We show this on an extension of an existing state-of-theart speech recognition system, a modular MS-TDNN. The acoustic and visual speech data is preclassified in two separate front-end phoneme TDNNs and combined to acousticvisual hypotheses for the Dynamic Time Warping algorithm. This is shown on a connected word recognition problem, the notoriously difficult letter spelling task. With speech-reading we could reduce the error rate up to half of the error rate of the pure acoustic recognition. I. INTRODUCTION Recent development in the design of human computer interfaces (HCI) requests a new field of research: MultiModal Recognition. There are strong research efforts in all kind of alternative computer input methods on the way, like speech input, or visual input, but usually all different recognition methods are treated sep...

Citations

2304	Learning Internal Representations by Error Propagation - Rumelhart, Hinton, et al. - 1986
293	Phoneme recognition using time-delay neural networks - Waibel, Hanaazawa, et al. - 1989
228	Neural Network Perception for Mobile Robot Guidance - Pomerleau - 1992
75	The use of a Onestage Dynamic Programming Algorithm for Connected Word Recognition - Ney - 1984
51	An improved automatic lipreading system to enhance speech recognition - Petajan, Bischoff, et al. - 1988
49	E.: Neural Network lip reading system for improved speech recognition - Stork, Wolff, et al. - 1992
47	A Novel Objective Function for Improved Phoneme Recognition Using Time-Delay Neural Networks - Hampshire, Waibel - 1990
35	Lipreading: Automatic visual recognition of spoken words - Pentland, Mase
27	Integration of acoustic and visual speech signals using neural networks - Yuhas, Goldstein, et al. - 1989
22	Multi-state time delay neural networks for continuous speech recognition - Haffner, Waibel - 1991
22	Connected Letter Recognition with a Multi-- State Time Delay - Hild, Waibel
2	Tahar Lallouache, Tayeb Mohamadi, and Christian Abry. A Set of French Visemes for Visual Speech Synthesis. Talking Machines: Theories, Models, and Designs - Benoit - 1992
2	Hearing by Eye: The Psychology fo Lipreading. Lawrence Erlbaum - Dodd, Campbell - 1987
2	Confusion among visually perceived consonants - Fischer - 1968
2	Facetracking Using a Template Based Approach. Personal Communication - Rander
2	Integrating Time Alignment and Connectionist Networks for High Performance Continuous Speech Recognition - Haffner, Frannzini, et al. - 1991