Toolglass ™ widgets are new user interface tools that can appear, as though on a transparent sheet of glass, between an application and a traditional cursor. They can be positioned with one hand while the other positions the cursor. The widgets provide a rich and concise vocabulary for operating on application objects. These widgets may incorporate visual filters, called Magic Lens™ filters, that modify the presentation of application objects to reveal hidden information, to enhance data of interest, or to suppress distracting information. Together, these tools form a see-through interface that offers many advantages over traditional controls. They provide a new style of interaction that better exploits the user’s everyday skills. They can reduce steps, cursor motion, and errors. Many widgets can be provided in a user interface, by designers and by users, without requiring dedicated screen space. In addition, lenses provide rich context-dependent feedback and the ability to view details and context simultaneously. Our widgets and lenses can be combined to form operation and viewing macros, and can be used over multiple applications.

Four techniques for performing a compound drawing/color selection task were studied: a unimanual technique, a bimanual technique where different hands controlled independent subtasks, and two other bimanual techniques in which the action of the right hand depended on that of the left hand. We call this latter class of two-handed technique “asymmetric dependent,” and predict that because tasks of this sort most closely conform to bimanual tasks in the everyday world, they would give rise to the best performance. Results showed that one of the asymmetric bimauual techniques, called the Toolglass technique, did indeed give rise to the best overall performance. Reasons for the superiority of this technique are discussed in terms of their implications for design. These are contrasted with other kinds of two-handed techniques, and it is shown how, if designed inappropriately, two hands can be worse than one.

In this paper we present a system that electromagnetically tracks the positions and orientations of multiple wireless objects on a tabletop display surface. The system offers two types of improvements over existing tracking approaches such as computer vision. First, the system tracks objects quickly and accurately without susceptibility to occlusion or changes in lighting conditions. Second, the tracked objects have state that can be modified by attaching physical dials and modifiers. The system can detect these changes in realtime. We present several new interaction techniques developed in the context of this system. Finally, we present two applications of the system: chemistry and system dynamics simulation. Keywords Tangible user interface, interactive surface, object tracking, two-handed manipulation, system dynamics, augmented reality

In current interfaces, users select objects, apply operations, and change viewing parameters in distinct steps that require switching attention among several screen areas. Our See-Through Interface ™ software reduces steps by locating tools on a transparent sheet that can be moved over applications with one hand using a trackball, while the other hand controls a mouse cursor. The user clicks through a tool onto application objects, simultaneously selecting an operation and an operand. Tools may include graphical filters that display a customized view of application objects. Compared to traditional interactors, these tools save steps, require no permanent screen space, reduce temporal modes, apply to multiple applications, and facilitate customization. This paper presents a taxonomy of see-through tools that considers variations in each of the steps they perform. As examples, we describe particular see-through tools that perform graphical editing and text editing operations.

The small size of handheld computers provides the convenience of mobility at the expense of reduced screen space for display and interaction. Prior research [5, 6] has identified the value of spatially aware displays, in which a position-tracked display provides a window on a larger virtual workspace. This paper builds on that work by suggesting two-handed interaction techniques combining pen input with spatially aware displays. Enabling simultaneous navigation and manipulation yields the ability to create and edit objects larger than the screen and to drag and drop in 3-D. Four prototypes of the Peephole Display hardware were built, and several Peephole-augmented applications were written, including a drawing program, map viewer, and calendar. Multiple applications can be embedded into a personal information space anchored to the user's physical reference frame. A usability study with 24 participants shows that the Peephole technique can be more effective than current methods for navigating information on handheld computers.

A major difficulty in writing Single Display Groupware (co-present collaborative) applications is getting input from multiple devices. We introduce MID, a Java package that addresses this problem and offers an architecture to access advanced events through Java. In this paper, we describe the features, architecture and limitations of MID. We also briefly describe an application that uses MID to get input from multiple mice: KidPad. Keywords Single Display Groupware (SDG), Computer-Supported Cooperative Work (CSCW), Multiple Input Devices (MID), Multi-Modal Input, Java, DirectInput, Windows 98, Universal Serial Bus (USB), KidPad, Jazz, Pad++. INTRODUCTION Communication, collaboration, and coordination are brought to many people's desktops thanks to groupware applications such as Lotus Notes and Microsoft Exchange, some of the leading commercial products in the field of Computer-Supported Cooperative Work (CSCW). They help people collaborate when they are not in the same place at th...

This dissertation defines and explores Graspable User Interfaces, an evolution of the input mechanisms used in graphical user interfaces (GUIs). A Graspable UI design provides users concurrent access to multiple, specialized input devices which can serve as dedicated physical interface widgets, affording physical manipulation and spatial arrangements. Like conventional GUIs, physical devices function as “handles” or manual controllers for logical functions on widgets in the interface. However, the notion of the Graspable UI builds on current practice in a number of ways. With conventional GUIs, there is typically only one graphical input device, such as a mouse. Hence, the physical handle is necessarily “time-multiplexed,” being repeatedly attached and unattached to the various logical functions of the GUI. A significant aspect of the Graspable UI is that there can be more than one input device. Hence input control can then be “space-multiplexed.” That is, different devices can be attached to different functions, each independently (but possibly simultaneously) accessible. This, then affords the capability to take advantage of the

Previous research has shown that rotation and orientation of items plays three major roles during collaboration: comprehension, coordination and communication. Based on these roles of orientation and advice from kinesiology research, we have designed the Rotate’N Translate (RNT) interaction mechanism, which provides integrated control of rotation and translation using only a single touch-point for input. We present an empirical evaluation comparing RNT to a common rotation mechanism that separates control of rotation and translation. Results of this study indicate RNT is more efficient than the separate mechanism and better supports the comprehension, coordination and communication roles of orientation.

One of the recent trends in computer input is to utilize users' natural bimanual motor skills. This paper further explores the potential benefits of such two-handed input. We have observed that bimanual manipulation may bring two types of advantages to human-computer interaction: manual and cognitive. Manual benefits come from increased time-motion efficiency, due to the twice as many degrees of freedom simultaneously available to the user. Cognitive benefits arise as a result of reducing the load of mentally composing and visualizing the task at an unnaturally low level imposed by traditional unimanual techniques. Area sweeping was selected as our experimental task. It is representative of what one encounters, for example, when sweeping out the bounding box surrounding a set of objects in a graphics program. Such tasks can not be modelled by Fitts' Law alone (Fitts, 1954) and have not been previously studied in the literature. In our experiments, two bimanual techniques were compared with the conventional one-handed GUI approach. Both bimanual techniques employed the two-handed "stretchy" technique first demonstrated by Krueger (1983). We also incorporated the "Toolglass" technique introduced by Bier, Stone, Pier, Buxton and DeRose (1993). Overall, the bimanual techniques resulted in significantly faster performance than the status quo one-handed technique, and these benefits increased with the difficulty of mentally visualizing the task, supporting our bimanual cognitive advantage hypothesis. There was no significant difference between the two bimanual techniques. This study makes two types of contributions to the literature. First, practically we studied yet another class of transaction where significant benefits can be realized by applying bimanual techniques. Furth...

In face-to-face conversation humans frequently use deicfic gestures parallel to verbal descriptions for referent identification. Such a multimodal form of communication is of great importance for intelligent interfaces, because it simplifies and speeds up reference to objects in a visual context. Natural pointing behavior is very flexible, but possibly ambiguous or vague, so that without a careful analysis of the discourse context of a gesture there would be a high risk of reference failure. The subject of this paper is how the user and discourse models of an intelligent interface influence the comprehension and, production of natural language with coordinated pointing, and conversely how multimodal communication influences the user model and the discourse model.