The detection and handling of OOD (out-of-domain) user utterances are significant problems for spoken language systems. We approach these problems by applying an OOD detection framework, combining topic classification and in-domain verification. In this paper, we compare the performance of three topic classification modeling schemes: 1-vs-all, where a single classifier is trained for each topic; weighted 1-vs-all; and 1-vs-1, which combines multiple pair-wise classifiers. We also compare the performance of a linear discriminate verifier and nonlinear SVM-based verification. In an OOD detection task as a front-end for speech-to-speech translation, detection performance was comparable for all classification schemes, indicating that the simplest 1-vs-all approach is sufficient for this task. SVM-based in-domain verification was found to provide a significant reduction in detection errors compared to a linear discriminate model. However, when the training and testing scenarios differ, the SVM approach was not robust, while the linear discriminate model remained effective. 1.

Abstract — One significant problem for spoken language systems is how to cope with users ’ OOD (out-of-domain) utterances which cannot be handled by the back-end application system. In this paper, we propose a novel OOD detection framework, which makes use of the classification confidence scores of multiple topics and applies a linear discriminant model to perform in-domain verification. The verification model is trained using a combination of deleted interpolation of the in-domain data and minimumclassification-error training, and does not require actual OOD data during the training process, thus realizing high portability. When applied to the “phrasebook ” system, a single utterance read-style speech task, the proposed approach achieves an absolute reduction in OOD detection errors of up to 8.1 points (40% relative) compared to a baseline method based on the maximum topic classification score. Furthermore, the proposed approach realizes comparable performance to an equivalent system trained on both in-domain and OOD data, while requiring no OOD data during training. We also apply this framework to the “machineaided-dialogue” corpus, a spontaneous dialogue speech task, and extend the framework in two manners. First, we introduce topic clustering which enables reliable topic confidence scores to be generated even for indistinct utterances, and second, we implement methods to effectively incorporate dialogue context. Integration of these two methods into the proposed framework significantly improves OOD detection performance, achieving a further reduction in EER of 7.9 points. I.

The detection and handling of OOD (out-of-domain) user utterances are significant problems for spoken language systems. We have proposed a novel OOD detection framework, which makes use of classification confidence scores of multiple topics. In this paper, we extend this framework in order to handle natural language dialogue. Specifically, two issues are addressed. First, to effectively incorporate dialogue context, we investigate methods to combine multiple utterances at various stages of the OOD detection process. Second, to improve robustness on spontaneous speech, we introduce a topic clustering scheme which provides reliable topic classification confidence even for indistinct utterances. The system is evaluated on natural dialogue via the ATR speech-to-speech translation system, and a significant improvement in OOD detection accuracy was achieved by incorporating the two proposed techniques. 1.

Conventional confidence measures for assessing the reliability of ASR output are typically derived from “low-level ” information which is obtained during speech recognition decoding. In contract to these approaches, we propose a novel utterance verification scheme which incorporates confidence measures derived from “high-level ” knowledge sources. Specifically, we investigate two measures: in-domain confidence, the degree of match between the input utterance and the application domain of the back-end system, and discourse coherence, the consistency between consecutive utterances in a dialogue session. A joint verification confidence is generated by combining these two measures with an orthodox measure based on GPP (generalized posterior probability). The proposed verification scheme was evaluated on spontaneous dialogue via the ATR speech-tospeech translation system. The two proposed measures were effective in improving verification accuracy. 1.

The detection and handling of OOD (out-of-domain) user utterances are significant problems for spoken language systems. We approach these problems by applying an OOD detection framework, combining topic classification and in-domain verification. In this paper, we compare the performance of three topic classification modeling schemes: 1-vs-all, where a single classifier is trained for each topic; weighted 1-vs-all; and 1-vs-1, which combines multiple pair-wise classifiers. We also compare the performance of a linear discriminate verifier and nonlinear SVM-based verification. In an OOD detection task as a front-end for speech-to-speech translation, detection performance was comparable for all classification schemes, indicating that the simplest 1-vs-all approach is sufficient for this task. SVM-based in-domain verification was found to provide a significant reduction in detection errors compared to a linear discriminate model. However, when the training and testing scenarios differ, the SVM approach was not robust, while the linear discriminate model remained effective. 1.