Multimodal interfaces require effective parsing and understanding of utterances whose content is distributed across multiple input modes. Johnston 1998 presents an approach in which strategies for multimodal integration are stated declaratively using a unification-based grammar that is used by a multidimensional chart parser to compose inputs. This approach is highly expressive and supports a broad class of interfaces, but offers only limited potential for mutual compensation among the input modes, is subject to significant concerns in terms of computational complexity, and complicates selection among alternative multimodal interpretations of the input. In this paper, we present an alternative approach in which multimodal parsing and understanding are achieved using a weighted finite-state device which takes speech and gesture streams as inputs and outputs their joint interpretation. This approach is significantly more efficient, enables tight-coupling of multimodal understanding with speech recognition, and provides a general probabilistic framework for multimodal ambiguity resolution. 1

Introduction Finite-state machines have been extensively applied to many aspects of language processing including, speech recognition (Pereira and Riley, 1997; Riccardi et al., 1996), phonology (Kaplan and Kay, 1994; Kartunnen, 1991), morphology (Koskenniemi, 1984), chunking (Abney, 1991; Joshi and Hopely, 1997; Bangalore, 1997), parsing (Roche, 1999), and machine translation (Bangalore and Riccardi, 2000). In Johnston and Bangalore (2000) we showed how finite-state methods can be employed in a new and different task - parsing, integration, and understanding of multimodal input. Our approach addresses the particular case of multimodal input to a mobile device where the modes are speech and gestures made on the display with a pen, but has far broader application. The approach uses a multimodal grammar specification which is compiled into a finite-state device running on three tapes. This device takes as input a speech stream and a gesture stream and outputs their combined meaning

While much research on the hard problem of in-depth story understanding by computer was performed starting in the 1970s, interest shifted in the 1990s to information extraction and word sense disambiguation. Now that a degree of success has been achieved on these easier problems, I propose it is time to return to in-depth story understanding. In this paper I examine the shift away from story understanding, discuss some of the major problems in building a story understanding system, present some possible solutions involving a set of interacting understanding agents, and provide pointers to useful tools and resources for building story understanding systems.

. Experiments point at the need for a robust grammatical analysis on recognition results. A robust parsing can first, help distinguish the incorrect recognitions that are recoverable; second, produce a relevant analysis for the interpretation process. We review existing robust parsing techniques and propose an approach based on Lexicalized Tree Grammars. 1 Evidence from Recognition Data 1.1 Background We report experiments conducted on "Virtual Speaker", an application developped at Thomson-CSF Corporate Research Lab., which aims at helping users chose their TV-program through a dialogue interface. We tested different speakers and two benchs of approximatly 50 sentences each, on three SR (Speech Recognition) systems 1 presented in Table 1, where CFG stands for Context Free Grammar and LM stands for Language Model. Table 1. tested SR systems System technology availability distributor origin Nuance flr CFG-driven market Nuance Communication SRI Decipher HTK-HAPI flr LM-driven mar...

Abstract not found

who laid the foundations for this research iii Acknowledgements These past five and a half years of doctoral studies at New York University have constituted a personally transformative experience (and I am claiming this independently of the jazz clubs, concert halls and vibrant community populating the greater Greenwich Village area). During these years, I have benefited from countless contributions that are impossible to acknowledge in a few lines. I will limit myself to mentioning a few individuals who directly enabled this work, hoping to eventually have the opportunity to contribute to someone else’s development in return. I would like to immensely thank my adviser, Prof. Yann LeCun, for providing me with resources, guidance, and freedom to pursue my research. Merci beaucoup pour avoir cru en moi, Yann. Yann LeCun’s lab is an intellectual hub with connections far beyond the field of Machine Learning, and therefore a very exciting research environment.