We investigate using the PARADISE framework to develop predictive models of system performance in our spoken dialogue tutoring system. We represent performance with two metrics: user satisfaction and student learning. We train and test predictive models of these metrics in our tutoring system corpora. We predict user satisfaction with 2 parameter types: 1) system-generic, and 2) tutoringspecific. To predict student learning, we also use a third type: 3) user affect. Alhough generic parameters are useful predictors of user satisfaction in other PAR-ADISE applications, overall our parameters produce less useful user satisfaction models in our system. However, generic and tutoring-specific parameters do produce useful models of student learning in our system. User affect parameters can increase the usefulness of these models. 1

We are exploring the differences between expert and less expert tutors with two goals: cognitive (what does tutoring tell us about learning) and applied (which features of tutoring dialogues should be included in interfaces to ITSs). We report results from human tutoring dialogues where an expert tutor was compared to less expert tutors. We also report results from a comparison among four versions of an ITS, that vary in the degree and kind of feedback they provide. Our results establish upper and lower bounds for the effectiveness of tutoring interactions in our domain.

Two measures of lexical cohesion were developed and applied to a corpus of human-computer tutoring dialogs. For both measures, the amount of cohesion in the tutoring dialog was found to be significantly correlated to learning for students with below-mean pretest scores, but not for those with above-mean pre-test scores, even though both groups had similar amounts of cohesion. We also find that only cohesion between tutor and student is significant: the cohesiveness of tutor, or of student, utterances is not. These results are discussed in light of previous work in textual cohesion and recall.

We examine correlations between student learning and student acoustic-prosodic profiles, which prior research has shown to be predictive of emotional states. We compare these correlations in two corpora of spoken tutoring dialogues: a human-human corpus and a human-computer corpus. Our results suggest that rather than relying on emotion prediction models developed via the more labor-intensive method of manually labeling emotions, adaptive strategies for our spoken dialogue tutoring system can be developed based on observed acoustic-prosodic profiles that we hypothesize to be reflective of emotion. 1.

We study correlations between dialog content and learning in a corpus of human-computer tutoring dialogs. Using an online encyclopedia, we first extract domainspecific concepts discussed in our dialogs. We then extend previously studied shallow dialog metrics by incorporating content at three levels of granularity (word, turn and discourse) and also by distinguishing between students ’ spoken and written contributions. In all experiments, our content metrics show strong correlations with learning, and outperform the corresponding shallow baselines. Our word-level results show that although verbosity in student writings is highly associated with learning, verbosity in their spoken turns is not. On the other hand, we notice that content along with conciseness in spoken dialogs is strongly correlated with learning. At the turn-level, we find that effective tutoring dialogs have more content-rich turns, but not necessarily more or longer turns. Our discourse-level analysis computes the distribution of content across larger dialog units and shows high correlations when student contributions are rich but unevenly distributed across dialog segments.

Speech recognition errors have been shown to negatively correlate with user satisfaction in evaluations of task-oriented spoken dialogue systems. In the domain of tutorial dialogue systems, however, where the primary evaluation metric is student learning, there has been little investigation of whether speech recognition errors also negatively correlate with learning. In this paper we examine correlations between student learning and automatic speech recognition performance, in a corpus of dialogues collected with an intelligent tutoring spoken dialogue system. We examine numerous quantitative measures of speech recognition error, including rejection versus misrecognition errors, word versus sentence-level errors, and transcription versus semantic errors. Our results show that although many of our students experience problems with speech recognition, none of our measures negatively correlates with student learning. 1.

Abstract. A previously reported measure of dialog cohesion was extended to measure cohesion by counting semantic similarity (the repetition of meaning) as well as lexical reiteration (the repetition of words) cohesive ties. Adding semantic similarity ties improved the algorithm’s correlation with learning among high pre-testers in one of our corpora of tutoring dialogs, where the lexical reiteration measure alone had correlated only for low pre-testers. Counting cohesive ties which have increasing semantic distance increases the measure’s correlation with learning in that corpus. We also find that both directions of tie, student-to-tutor and tutor-to-student, are equally important in producing these correlations. Finally, we present evidence suggesting that the correlations we find may be with deeper “far transfer ” learning. 1

Abstract not found

Experimental research has shown that human users will converge with dialog systems along many dimensions of speech, including those of acoustic/prosodic features and lexical choice. Other results suggest that speech convergence may provide a variety of benefits to spoken dialog systems, such as an improved user model, increased ease of use, improved feelings of intimacy, and increased compliance on the part of the user. These potential benefits to dialog systems of generating or detecting convergence behaviors suggest the need for corpus studies of convergence, in addition to the experimental results. Here, we build on previous work to demonstrate corpus measures of lexical and acoustic/prosodic convergence. We show that these measures successfully distinguish randomized from naturally ordered data, and demonstrate both lexical and acoustic/prosodic priming effects in our corpus of human/human tutoring dialogs. 1

Abstract. Studies of one-on-one tutoring have found that expert tutoring is more effective than non-expert tutoring, but the reasons for its effectiveness are relatively unexplored. Since tutoring involves deep natural language interactions between tutor and student, we explore the differences between an expert and non-expert tutors through the analysis of individual dialogue moves, tutorial interaction patterns and multiutterance turns. Our results are a first step showing what behaviors constitute expertise and provide a basis for modeling effective tutorial language in intelligent tutoring systems. 1