This report presents a detailed analysis and review of NLP evaluation, in principle and in practice. Part 1 examines evaluation concepts and establishes a framework for NLP system evaluation. This makes use of experience in the related area of information retrieval and the analysis also refers to evaluation in speech processing. Part 2 surveys significant evaluation work done so far, for instance in machine translation, and discusses the particular problems of generic system evaluation. The conclusion is that evaluation strategies and techniques for NLP need much more development, in particular to take proper account of the influence of system tasks and settings. Part 3 develops a general approach to NLP evaluation, aimed at methodologically-sound strategies for test and evaluation motivated by comprehensive performance factor identification. The analysis throughout the report is supported by extensive illustrative examples. This work was carried out under the UK Science and Engineeri...

This article addresses the concern for effective web site design by means of the conceptualization and empirical validation of a web site user satisfaction construct. Based on IS success theory, hypermedia design theory, a qualitative exploratory pilot study, and a quantitative online critical incident technique, we introduce and define the construct of web site user satisfaction, explore its dimensionality, provide empirical validation of the construct and its underlying dimensionality, develop a standardized instrument with desirable psychometric properties for measuring WUS, and explore the measure's theoretical and practical application.

This report describes the current state of my research in software development environments. I argue in favour of strong support for project management, comprehension and navigation, and collaboration primarily based on experiences from developing large-scale industrial-strength applications. An underlying model of such an environment, named "Ragnarok", is outlined. A design and first prototype of important parts of Ragnarok is described as well as some results from initial experiments. Contents 1 Introduction 4 2 Motivation 4 2.1 A retrospective case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3 Contours 6 3.1 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 Aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2.1 Project management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2.2 Comprehension & Navigation . . . . . . . . . . . . . ....

. The issue of analyzing and conceptualizing cooperative work and its coordination has gained considerable impetus over the last two decades. The Coordination Mechanism framework falls within that body of work by addressing the functioning of artifacts as means of supporting the coordination of complex cooperative work. The present paper takes the framework outside the world of artifacts and presents an initial investigation of the framework's applicability to the analysis of spoken coordination. The main focus is on investigating to what extent the analysis of spoken coordination can benefit from the conceptualizations of cooperation and coordination that have been established on the basis of empirical studies of artifact-based coordination. The investigation was undertaken within the work domain of maritime operations; a domain characterized by highly complex and time critical cooperation and coordination. The analysis of maritime operations illustrates that the basic ana...

In this work we take the point of departure from two thoroughly conducted case studies, in which we have analysed certain knowledge work activities with following implications for design. Both studies clearly stated the call for a ICT-support when searching information and sharing knowledge. For example, sharing bookmarks among knowledge workers seemed to be a necessary and useful activity. Important and valuable bookmarks often mirror what kind of information and knowledge people are looking for, and also what kind of forum they usually go to on the world wide web. In this work we have developed a prototype, KnowMan, which manages bookmarks on the world wide web. The prototype is designed as a user-friendly "pad-interface" in which bookmarks can be collaboratively collected, shared and further explored and annotated by other knowledge workers. We end up with a general discussion about the possible effects of using KnowMan and what further design aspects that should be considered.

The issue of analyzing and conceptualizing cooperative work and its coordination has gained considerable impetus over the last two decades. The Coordination Mechanism framework falls within that body of work by addressing the functioning of artifacts as means of supporting the coordination of complex cooperative work. The present paper takes the framework outside the world of artifacts and presents an initial investigation of the frameworks applicability to the analysis of spoken coordination. The main focus is on investigating to what extent the analysis of spoken coordination can benefit from the conceptualizations of cooperation and coordination that have been established on the basis of empirical studies of artifact-based coordination. The investigation was undertaken within the work domain of maritime operations; a domain characterized by highly complex and time critical cooperation and coordination. The analysis of maritime operations illustrates that the basic analytical concepts underpinning the Coordination Mechanisms framework are also useful when addressing spoken coordination, and that central functional aspects of Coordination Mechanisms can also be identified in spoken coordination. The basic functional understanding of a coordination mechanism, as a means of stipulating and mediating coordination information, proved useful, whereas the restricted notion of physical artifacts as coordination means was, not surprisingly, too narrow for work settings mainly characterized by oral coordination. 1.