The World as your Palette" is our ongoing effort to design and develop tools to allow artists to create visual art projects with elements (specifically, the color, texture, and moving patterns) extracted directly from their personal objects and their immediate environment. Our tool called "I/O Brush" looks like a regular physical paintbrush, but contains a video camera, lights, and touch sensors. Outside of the drawing canvas, the brush can pick up colors, textures, and movements of a brushed surface. On the canvas, artists can draw with the special "ink" they just picked up from their immediate environment. We describe the evolution and development of our system, from kindergarten classrooms to an art museum, as well as the reactions of our users to the growing expressive capabilities of our brush, as an iterative design process. Keywords Drawing, Children, Storytelling, Interaction Design, Tangible User Interfaces, Usability Research, User Experience, User Interface Design, User Research, User-Centered Design / Human-Centered Design, Visual Design Categories and subject descriptors Copyright is held by the author/owner(s). CHI 2--7, 2005, Portland, Oregon, USA. ACM 1-59593-002-7/05/0004.

Tangible user interfaces (TUIs) provide physical form to digital information and computation, facilitating the direct manipulation of bits. Our goal in TUI development is to empower collaboration, learning, and design by using digital technology and at the same time taking advantage of human abilities to grasp and manipulate physical objects and materials. This paper discusses a model of TUI, key properties, genres, applications, and summarizes the contributions made by the Tangible Media Group and other researchers since the publication of the first Tangible Bits

Mediated reality describes the concept of filtering our vision of reality, typically using a head-worn video mixing display. In this paper, we propose a generalized concept and new tools for interactively mediated reality. We present also our first prototype system for painting, grabbing and glueing together real and virtual elements. 1.

We present an interactive approach to non-photorealistic rendering that contrasts with the standard black box character of previous rendering techniques in that observation and interaction take place during rendering. Our technique is based on the idea of approaching non-photorealistic rendering by modeling with rendering primitives. This new approach supports interruption, tweaking, manipulation, and re-direction of the rendering as it develops. While we draw upon computational support for primitive placement to avoid having to painstakingly place each pixel, we limit the computational influence to enable freedom of interaction with the elements. We implement this new paradigm in a stroke-based rendering application using a stack of interaction buffers to store attributes of the primitives during the rendering. By manipulating the data in these buffers we affect the behavior of strokes on the canvas. This allows us to create and adjust images in non-photorealistic styles such as painterly rendering, pointillism, and decorative mosaics at interactive frame rates.

Digital Manipulatives embed computation in familiar children’s toys and provide means for children to design behavior. Some systems use “record and play ” as a form of programming by demonstration that is intuitive and easy to learn. With others, children write symbolic programs with a GUI and download them into a toy, an approach that is conceptually extensible, but is inconsistent with the physicality of educational manipulatives. The challenge we address is to create a tangible interface that can retain the immediacy and emotional engagement of “record and play ” and incorporate a mechanism for real time and direct modulation of behavior during program execution. We introduce the Backpacks, modular physical components that children can incorporate into robotic creations to modulate frequency, amplitude, phase and orientation of motion recordings. Using Backpacks, children can investigate basic kinematic principles that underlie why their specifi c creations exhibit the specifi c behaviors they observe. We demonstrate that Backpacks make tangible some of the benefi ts of symbolic abstraction, and introduce sensors, feedback and behavior modulation to the record and play paradigm. Through our review of user studies with children ages 6-15, we argue that Backpacks extend the conceptual limits of record and play with an interface that is consistent with both the physicality of educational manipulatives and the local-global systems dynamics that are characteristic of complex robots.

We introduce Jabberstamp, the first tool that allows children to synthesize their drawings and voices. To use Jabberstamp, children create drawings, collages or paintings on normal paper. They press a special rubber stamp onto the page to record sounds into their drawings. When children touch the marks of the stamp with a small trumpet, they can hear the sounds playback, retelling the stories they created. We describe our design process and analyze the mechanism between the act of drawing and the one of telling, defining interdependencies between the two activities. In a series of studies, children ages 4-8 use Jabberstamp to convey meaning in their drawings. The system allows collaboration among peers at different developmental levels. Jabberstamp compositions reveal children’s narrative styles and their planning strategies. In guided activities, children develop stories by situating sound recording in their drawing, which suggests future opportunities for hybrid voice–visual tools to support children’s emergent literacy.

We present Picture This! a new input device embedded in children’s toys for video composition. It consists of a new form of interaction for children’s capturing of storytelling with physical artifacts. It functions as a video and storytelling performance system in that children craft videos with and about character toys as the system analyzes their gestures and play patterns. Children’s favorite props alternate between characters and cameramen in a film. As they play with the toys to act out a story, they conduct film assembly. We position our work as ubiquitous computing that supports children’s tangible interaction with digital materials. During user testing, we observed children ages 4 to 10 playing with Picture This!. We assess to what extent gesture interaction with objects for video editing allows children to explore visual perspectives in storytelling. A new genre of Gesture Object Interfaces as exemplified by Picture This relies on the analysis of gestures coupled with objects to represent bits. Author Keywords Video, gesture object interfaces, children, play, storytelling.

We explore the use of hand postures to interact with stroke-based rendering (SBR) on touch-sensitive large displays. In contrast to traditional WIMP interfaces, we allow people to directly engage with and influence a rendering. Our system allows the creation of new stroke primitives as well as provides mechanisms to distribute and then manipulate them on the canvas. We offer a set of natural mappings from hand postures to rendering parameterizations. The resulting system allows an intuitive exploration of SBR without the need for traditional desktop interfaces.

We introduce two new interaction techniques for creating and interacting with non-photorealistic images using strokebased rendering. We provide bimanual control of a large interactive canvas through both remote pointing and direct touch. Remote pointing allows people to sit and interact at a distance with an overview of the entire display, while direct-touch interaction provides more precise control. We performed a user study to compare these two techniques in both a controlled setting with constrained tasks and an exploratory setting where participants created their own painting. We found that, although the direct-touch interaction outperformed remote pointing, participants had mixed preferences and did not consistently choose one or the other to create their own painting. Some participants also chose to switch between techniques to achieve different levels of precision and control for different tasks. (a) Remote pointing with Wiimote.

Abstract. The paper presents a novel approach to collecting, editing and performing visual and sound clips in real time. The cumbersome process of capturing and editing becomes fluid in the improvisation of a story, and accessible as a way to create a final movie. It is shown how a graphical interface created for video production informs the design of a tangible environment that provides a spontaneous and collaborative approach to video creation, selection and sequencing. Iterative design process, participatory design sessions and workshop observations with 10-12 year old users from Sweden and Ireland are discussed. The limitations of interfacing video capture, editing and publication in a self-contained platform are addressed.