. Micro-task markets, such as Amazon’s Mechanical Turk, offer a potential paradigm for engaging a large number of users for low time and monetary costs. Here we investigate the utility of a micro-task market for collecting user measurements, and discuss design considerations for developing remote micro

the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation benchmarking campaign and this allows us to discuss whether such campaigns are a good thing or a bad thing. There are arguments for and against these campaigns and we present some

Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa statistic. We discuss what is wrong with reliability measures as they are currently used for discourse and dialogue work in computational linguistics and cognitive science, and argue that we would be better off as a field adopting techniques from content analysis. 1

…Truth [is] being involved in an eternal conversation about things that matter, conducted with passion and discipline…truth is not in the conclusions so much as in the process of conversation itself…if you want to be in truth you must be in conversation. Parker Palmer

A useful starting point for designing advanced graphical user interjaces is the Visual lnformation-Seeking Mantra: overview first, zoom and filter, then details on demand. But this is only a starting point in trying to understand the rich and varied set of information visualizations that have been

Reliability is even more important in object-oriented programming than elsewhere. This article shows how to reduce bugs by building software components on the basis of carefully designed contracts. 40 s object-oriented techniques steadily gain ground in the world of software development. users

Data generated as a side effect of game play also solves computational problems and trains AI algorithms.

This individual differences study examined the separability of three often postu-lated executive functions—mental set shifting (‘‘Shifting’’), information updating and monitoring (‘‘Updating’’), and inhibition of prepotent responses (‘‘Inhibi-tion’’)—and their roles in complex ‘‘frontal lobe

This paper presents a methodology for agent-oriented analysis and design. The methodology is general, in that it is applicable to a wide range of multi-agent systems, and comprehensive, in that it deals with both the macro-level (societal) and the micro-level (agent) aspects of systems

The World Wide Web has succeeded in large part because its software architecture has been designed to meet the needs of an Internet-scale distributed hypermedia system. The modern Web architecture emphasizes scalability of component interactions, generality of interfaces, independent deployment