Common data-driven evaluation metrics for speech understanding systems are based on automatically comparing sequences of slot--value pairs by dynamic programming (DP) matching. However, for complex hierarchical language models sequence matching based metrics don't seem appropriate as they cannot fully capture structural similarities. For this task we propose a novel evaluation metric, the tree node accuracy. Our approach is founded on a DP-style algorithm that computes the minimum edit distance between pairs of ordered labeled trees and includes the sequence matching problem as a special case. We also extended the basic scheme for our task to support trees consisting of different categories of tree nodes. Experiments carried out on several semantic models confirm that the tree matching based approach displays greater flexibility than conventional sequence matching based metrics and is especially suited for complex hierarchical models.

One-stage decoding as an integration of speech recognition and linguistic analysis into one probabilistic process is an interesting trend in speech research. In this paper, we present a simple one-stage decoding scheme that can be realised without the implementation of a specialized decoder, nor the use of complex language models. Instead, we reduce an HMMbased semantic analysis to the problem of deriving annotated versions of the conventional language model, while the acoustic model remains unchanged. We present experiments with the ATIS corpus (Price, 1990) in which the performance of the one-stage method is shown to be comparable with the traditional two-stage approach, while requiring a significantly smaller increase in language model size. 1