We present a technique for evaluating the usability and e#ectiveness of ambient displays. Ambient displays are abstract and aesthetic peripheral displays portraying non-critical information on the periphery of a user's attention. Although many innovative displays have been published, little existing work has focused on their evaluation, in part because evaluation of ambient displays is di#cult and costly. We adapted a low-cost evaluation technique, heuristic evaluation, for use with ambient displays. With the help of ambient display designers, we defined a modified set of heuristics. We compared the performance of Nielsen's heuristics and our heuristics on two ambient displays. Evaluators using our heuristics found more, severe problems than evaluators using Nielsen's heuristics. Additionally, when using our heuristics, 3-5 evaluators were able to identify 40-60% of known usability issues. This implies that heuristic evaluation is an e#ective technique for identifying usability issues with ambient displays.

Note that the text that appeared in printed journal contains very minor typographic and grammatical corrections that do not appear in this version. SpeechSkimmer:

This paper suggests a framework for understanding the roles that children can play in the technology design process, particularly in regards to designing technologies that support learning. Each role, user, tester, informant, and design partner has been defined based upon a review of the literature and my lab’s own research experiences. This discussion does not suggest that any one role is appropriate for all research or development needs. Instead, by understanding this framework the reader may be able to make more informed decisions about the design processes they choose to use with children in creating new technologies. This paper will present for each role a historical overview, research and development methods, as well as the strengths, challenges, and unique contributions associated with children in the design process.

Researchers in Computer Supported Cooperative Work have recently developed discount evaluation methods for shared-workspace groupware. Most discount methods rely on some understanding of the context in which the groupware systems will be used, which means that evaluators need to model the tasks that groups will perform. However, existing task analysis schemes are not well suited to the needs of groupware evaluation: they either do not deal with collaboration issues, do not use an appropriate level of analysis for concrete assessment of usability in interfaces, or do not adequately represent the variability inherent in group work. To fill this gap, we have developed a new modeling technique called Collaboration Usability Analysis. CUA focuses on the teamwork that goes on in a group task rather than the taskwork. To enable closer links between the task representation and the groupware interface, CUA grounds each collaborative action in a set of group work primitives called the mechanics of collaboration. To represent the range of ways that a group task can be carried out, CUA allows variable paths through the execution of a task, and allows alternate paths and optional tasks to be modeled. CUA’s main contribution is to provide evaluators with a framework in which they can simulate the realistic use of a groupware system

Research into the cognitive aspects of programming originated in the study of professional programmers (whether experts or students). Even "end-user" programmers in previous studies have often worked in organizations where programming is recognized to be demanding professional work- the term 'vower-user" recognizes this technical kudos. But as personal computers become widespread, and most new domestic appliances incorporate microprocessors, many people are engaging in programming-like activities in domestic or non-professional contexts. Such users often have less motivation and more obstacles to programming, meaning that they may be unlikely even to take the first steps. This paper analyses the generic nature of those first steps, and identifies the cognitive demands that characterize them. On the basis of this analysis we propose the Attention Investment model, a cognitive model of programming that offers a consistent account of all programming behaviour, from professionals to end-users.

Field research methods are useful in the many aspects of Human-Computer Interaction research, including gathering user requirements, understanding and developing user models, and new product evaluation and iterative design. Due to increasingly short product realization cycles, there has been growing interest in more time efficient methods, including rapid prototyping and various usability inspection techniques. This paper will introduce "rapid ethnography, " which is a collection of field methods intended to provide a reasonable understanding of users and their activities given significant time pressures and limited time in the field. The core elements include limiting or constraining the research focus and scope, using key informants, capturing rich field data by using multiple observers and interactive observation techniques, and collaborative qualitative data analysis. A short case study illustrating the important characteristics of rapid ethnography will also be presented.

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)—3 of the most widely used UEMs—suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. Areview of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5 % to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations. Morten Hertzum was supported by a grant from the Danish National Research Foundation. We thank Iain Connell for providing us with additional data from Connell and Hammond (1999), Hilary Johnson for access to the data set from Dutt, Johnson, and Johnson (1994), and Rolf Molich for making the data set from Molich et al. (1999) available on the Web

Good real time groupware products are hard to develop, in part because evaluating their support for the basic activities of teamwork is difficult and costly. To address this problem, we are developing discount evaluation methods that look for groupware-specific usability problems. In a previous paper, we detailed a new set of usability heuristics that evaluators can use to inspect shared workspace groupware to see how they support for teamwork. We wanted to determine whether the new heuristics could be integrated into a low-cost methodology that parallels Nielsen’s traditional heuristic evaluation (HE). To this end, we examined 27 evaluations of two shared workspace groupware systems and analysed the inspectors ’ relative performance and variability. Similar to Nielsen’s findings for traditional HE, individual inspectors discovered about a fifth of the total known teamwork problems, and that there was only modest overlap in the problems they found. Groups of three to five inspectors would report about 40– 60 % of the total known teamwork problems. These results suggest that heuristic evaluation using our groupware heuristics can be an effective and efficient method for identifying teamwork problems in shared workspace groupware systems.

13.44> . Drew Kessler for help with the SVE toolkit . The Virtual Environments group at Georgia Tech . The numerous experimental subjects who volunteered their time . Dawn Bowman iv TABLE OF CONTENTS Introduction ..................................................................... ................. 1 1.1 Motivation ..................................................................... ...............1 1.2 Definitions.......................................................... ..........................4 1.3 Problem Statement............................................................ ...............6 1.4 Scope of the Research............................................................. ..........7 1.5 Hypotheses........................................................... ........................8 1.6 Contributions........................................................ .....

The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. In this article, we present a practical discussion of factors, comparison criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We highlight specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison. 1.