There is a growing interest in developing technologies for creating interactive dramas [13, 22]. Evaluating them, however, remains an open research problem. In this paper, we present a method for evaluating the technical and design approaches employed in a conversation-centered interactive drama. This method correlates players ’ subjective experience during conversational breakdowns, captured using retrospective protocols, with the corresponding AI processing in the input language understanding and dialog management subsystems. The methodology is employed to analyze conversation breakdowns in the interactive drama Façade. We find that the narrative cues offered by an interactive drama, coupled with believable character performance, can allow players to interpretively bridge system limitations and avoid experiencing a conversation breakdown. Further, we find that, contrary to standard practice for task-oriented conversation systems, using shallowly understood information as part of the system output hampers the player experience in an interactive drama.