Graphics have been used since ancient times to portray things that are inherently spatiovisual, like maps and building plans. More recently, graphics have been used to portray things that are metaphorically spatiovisual, like graphs and organizational charts. The assumption is that graphics can facilitate comprehension, learning, memory, communication and inference. Assumptions aside, research on static graphics has shown that only carefully designed and appropriate graphics prove to be beneficial for conveying complex systems. Effective graphics conform to the Congruence Principle according to which the content and format of the graphic should correspond to the content and format of the concepts to be conveyed. From this, it follows that animated graphics should be effective in portraying change over time. Yet the research on the efficacy of animated over static graphics is not encouraging. In cases where animated graphics seem superior to static ones, scrutiny reveals lack of equivalence between animated and static graphics in content or procedures; the animated graphics convey more information or involve interactivity. Animations of events may be ineffective because animations violate the second principle of good graphics, the Apprehension Principle, according to which graphics should be accurately perceived and appropriately conceived. Animations are often too complex or too fast to be accurately perceived. Moreover, many continuous events are conceived of as sequences of discrete steps. Judicious use of interactivity may overcome both these disadvantages. Animations may be more effective than comparable static graphics in situations other than conveying complex systems, for example, for real time reorientations in time and space.

Abstract. Traditionally, depictions and descriptions have been seen as complementary; depictions have been preferred to convey iconic or metaphorically iconic information whereas descriptions have been preferred for abstract information. Both are external representations designed to complement human memory and information processing. We have found the same underlying structure and semantics for route maps and route directions. Here we find that limited schematic map and direction toolkits are sufficient for constructing directions, supporting the possibility of automatic translation between them.

External representations such as diagrams, sketches, charts, graphs and scribbles on napkins play facilitatory roles in inference, problem-solving and understanding (e.g. [1],[2],[3],[4],[5],[6],[7],[8],[9]). How does the externality and visibility of representations facilitate inference and problem-solving? One benefit of external representations is on memory. They reduce working memory load by providing external tokens for the elements that must otherwise be kept in mind. This frees working memory to perform mental calculations on the elements rather than both keeping elements in mind and operating on them [2],[9]. External representations also serve as visuo-spatial retrieval cues for long term memory, evoking relevant information that might not otherwise be retrieved. Another benefit of external representations is to promote discovery and inference, both visuo-spatial and metaphorical. Perceptual judgements about size, distance, and direction are easily made from external representations (e.g.[4]). In a Venn diagram, set relations such as inclusion are abstractly mapped onto visuo-spatial diagrammatic features, enabling direct perceptual calculation. Visuo-spatial features such as proximity, connectivity, and alignment provide useful hints to selection of appropriate inference

Sketches are produced in many domains to communicate with self or others. They are a kind of external representation serving as a cognitive tool to augment memory and information processing by relieving the mind of some of those burdens. Sketches schematize. They do not portray reality; rather they convey conceptions of reality. Sketches use elements and spatial relations on paper to represent elements and spatial or abstract relations in the domain of interest. They incorporate relevant information and omit irrelevant. As such, they reveal people's conceptions of domains.. An analysis of sketches of routes, for example, reveals that they segment routes around action points, typically turns. Order of drawing reflects mental organization of the domain. From sketches, novices extract structural information about the spatial relations among elements. Experts are also able to extract functional information, information that must be inferred from rather than directly extracted from sketches.

In this paper, we describe an interface that demonstrates spatial intelligence. This interface, an embodied conversational kiosk, builds on research in embodied conversational agents (ECAs) and on information displays in mixed reality and kiosk format. ECAs leverage people's abilities to coordinate information displayed in multiple modalities, particularly information conveyed in speech and gesture. Mixed reality depends on users' interactions with everyday objects that are enhanced with computational overlays. We describe an implementation, MACK (Media lab Autonomous Conversational Kiosk), an ECA who can answer questions about and give directions to the MIT Media Lab's various research groups, projects and people. MACK uses a combination of speech, gesture, and indications on a normal paper map that users place on a table between themselves and MACK. Research issues involve users' differential attention to hand gestures, speech and the map, and how reference using these modalities can be fused in input and generation.

Within psychology, at least two research communities study spatial cognition. One community studies systematic errors in spatial memory and judgement, accounting for them as a consequence of and clue to normal perceptual and cognitive processing. The other community studies navigation in real space, isolating the contributions of various sensory cues and sensori-motor systems to successful navigation. The former group emphasizes error, the latter, selective mechanisms, environmental or evolutionary, that produce fine-tuned correct responses. How can these approaches be reconciled and integrated? First, by showing why errors are impervious to selective pressures. The schematization that leads to errors is a natural consequence of normal perceptual and cognitive processes; it is inherent to the construction of mental spaces and to using them to make judgments in limited capacity working memory. Selection can act on particular instances of errors, yet it is not clear that selection can act on the general mechanisms that produce them. Next, in the wild, there are a

In this paper, we describe an embodied conversational kiosk that builds on research in embodied conversational agents (ECAs) and on information displays in mixed reality and kiosk format in order to display spatial intelligence. ECAs leverage people’s abilities to coordinate information displayed in multiple modalities, particularly information conveyed in speech and gesture. Mixed reality depends on users ’ interactions with everyday objects that are enhanced with computational overlays. We describe an implementation, MACK (Media lab Autonomous Conversational Kiosk), an ECA who can answer questions about and give directions to the MIT Media Lab’s various research groups, projects and people. MACK uses a combination of speech, gesture, and indications on a normal paper map that users place on a table between themselves and MACK. Research issues involve users’ differential attention to hand gestures, speech and the map, and flexible architectures for Embodied Conversational Agents that allow these modalities to be fused in input and generation.

Design typically relies on diagrams to offload memory and information processing and to promote discovery and inferences. Design of information systems, in contrast to design of buildings and products, depends on topological connectivity rather than Euclidean distance. Understanding graph topology and manipulating graphs are essential skills in the design of information systems, because graph manipulation facilitates the refinement of designs and the generation of alternative designs. Here, we found that students of systems design have difficulties interpreting diagrams, revealing two biases, a sequential bias and a reading order bias. The results have implications for teaching as well as diagram design. Design Computing and Cognition DCC’08. J.S. Gero and

Communicators explained how to assemble an object or how to get from A to B to recipients, using speech and gesture freely, or restricted to gesture alone. Explanations were examined for uses of gesture and speech in conveying semantic content as well as narrative structure. For both tasks, communicators who explained using only gestures performed better than those using gestures and speech. Recipients also learned both tasks better after watching explanations that only used gestures. Specific gestures were related to specific contents of memory, suggesting that gestures change the nature of thought. Gestures seem to be effective for several reasons: they preferentially call attention to certain information, they bear iconic or metaphoric resemblance to the ideas they convey, and they embody and situate knowledge.

We investigate the use of external visual representations in probability problem solving. Twenty-six students enrolled in an introductory statistics course for social sciences graduate students (post-baccalaureate) solved eight probability problems in a structured interview format. Results show that students spontaneously use selfgenerated external visual representations while solving probability problems. The types of visual representations used include: reorganization of the given information, pictures, novel schematic representations, trees, outcome listings, contingency tables, and Venn diagrams. The frequency of use of each of these different external visual representations depended on the type of probability problem being solved. We interpret these findings as showing that problem solvers attempt to select representations appropriate to the problem structure, and that the appropriateness of the representation is determined by the problem’s underlying schema.