Toolglass ™ widgets are new user interface tools that can appear, as though on a transparent sheet of glass, between an application and a traditional cursor. They can be positioned with one hand while the other positions the cursor. The widgets provide a rich and concise vocabulary for operating on application objects. These widgets may incorporate visual filters, called Magic Lens™ filters, that modify the presentation of application objects to reveal hidden information, to enhance data of interest, or to suppress distracting information. Together, these tools form a see-through interface that offers many advantages over traditional controls. They provide a new style of interaction that better exploits the user’s everyday skills. They can reduce steps, cursor motion, and errors. Many widgets can be provided in a user interface, by designers and by users, without requiring dedicated screen space. In addition, lenses provide rich context-dependent feedback and the ability to view details and context simultaneously. Our widgets and lenses can be combined to form operation and viewing macros, and can be used over multiple applications.

We introduce the concept of Graspable User Interfaces which allow direct control of electronic or virtual objects through physical handles for control. These physical artifacts are essentially new input devices which can be tightly coupled or “attached ” to virtual objects for manipulation or for expressing action (e.g., to set parameters or for initiating processes). We present three steps in the development of these ideas. First, as a note on research methodology, we outline a series of exploratory studies that were conducted. Secondly, we describe a prototype system called "Bricks " and a sample application, GraspDraw, which was developed to investigate the Graspable UI concepts and to design new one- and twohanded interaction techniques. The physical artifacts, or bricks, operate on top of a large horizontal display surface known as the ActiveDesk. Finally, we conclude by presenting a design space for Bricks which lay the foundation for further exploring and developing graspable user interfaces.

This paper explores a user interface technique which augments an immersive head tracked display with a hand-held miniature copy of the virtual environment. We call this interface technique the Worlds in Miniature (WIM) metaphor. In addition to the first-person perspective offered by a virtual reality system, a World in Miniature offers a second dynamic viewport onto the virtual environment. Objects may be directly manipulated either through the immersive viewport or through the three-dimensional viewport offered by the WIM. In addition to describing object manipulation, this paper explores ways in which Worlds in Miniature can act as a single unifying metaphor for such application independent interaction techniques as object selection, navigation, path planning, and visualization. The WIM metaphor offers multiple points of view and multiple scales at which the user can operate, without requiring explicit modes or commands. Informal user observation indicates that users adapt to the ...

Four techniques for performing a compound drawing/color selection task were studied: a unimanual technique, a bimanual technique where different hands controlled independent subtasks, and two other bimanual techniques in which the action of the right hand depended on that of the left hand. We call this latter class of two-handed technique “asymmetric dependent,” and predict that because tasks of this sort most closely conform to bimanual tasks in the everyday world, they would give rise to the best performance. Results showed that one of the asymmetric bimauual techniques, called the Toolglass technique, did indeed give rise to the best overall performance. Reasons for the superiority of this technique are discussed in terms of their implications for design. These are contrasted with other kinds of two-handed techniques, and it is shown how, if designed inappropriately, two hands can be worse than one.

www.dgp.toronto.edu Recent advances in sensing technology have enabled a new generation of tabletop displays that can sense multiple points of input from several users simultaneously. However, apart from a few demonstration techniques [17], current user interfaces do not take advantage of this increased input bandwidth. We present a variety of multifinger and whole hand gestural interaction techniques for these displays that leverage and extend the types of actions that people perform when interacting on real physical tabletops. Apart from gestural input techniques, we also explore interaction and visualization techniques for supporting shared spaces, awareness, and privacy. These techniques are demonstrated within a prototype room furniture layout application, called RoomPlanner.

Manipulation in immersive virtual environments is difficult partly because users must do without the haptic contact with real objects they rely on in the real world to orient themselves and their manipulanda. To compensate for this lack, we propose exploiting the one real object every user has in a virtual environment, his body. We present a unified framework for virtual-environment interaction based on proprioception, a person's sense of the position and orientation of his body and limbs. We describe three forms of body-relative interaction: . Direct manipulation---ways to use body sense to help control manipulation . Physical mnemonics---ways to store/recall information relative to the body . Gestural actions---ways to use body-relative actions to issue commands Automatic scaling is a way to bring objects instantly within reach so that users can manipulate them using proprioceptive cues. Several novel virtual interaction techniques based upon automatic scaling and our proposed fr...

Reading frequently involves not just looking at words on a page, but also underlining, highlighting and commenting, either on the text or in a separate notebook. This combination of reading with critical thinking and learning is called active reading [2]. To explore the premise that computation can enhance active reading we have built the XLibris "active reading machine." XLibris uses a commercial high-resolution pen tablet display along with a paper-like user interface to support the key affordances of paper for active reading: the reader can hold a scanned image of a page in his lap and mark on it with digital ink. To go beyond paper, XLibris monitors the free-form ink annotations made while reading, and uses these to organize and to search for information. Readers can review, sort and filter clippings of their annotated text in a "Reader's Notebook." XLibris also searches for material related to the annotated text, and displays links to similar documents unobtrusively in the margin. XLibris demonstrates that computers can help active readers organize and find information while retaining many of the advantages of reading on paper. Keywords Paper-like user interface, reading online, affordances of paper, pen computing, dynamic hypertext, document metaphor, information retrieval

This article introduces a new interaction model called Instrumental Interaction that extends and generalizes the principles of direct manipulation. It covers existing interaction styles, including traditional WIMP interfaces, as well as new interaction styles such as two-handed input and augmented reality. It defines a design space for new interaction techniques and a set of properties for comparing them. Instrumental Interaction describes graphical user interfaces in terms of domain objects and interaction instruments. Interaction between users and domain objects is mediated by interaction instruments, similar to the tools and instruments we use in the real world to interact with physical objects. The article presents the model, applies it to describe and compare a number of interaction techniques, and shows how it was used to create a new interface for searching and replacing text. Keywords Interaction model, WIMP interfaces, direct manipulation, post-WIMP interfaces, instrumental ...

This paper describes the introduction of a new interaction paradigm to augmented reality applications. The everyday tool handling experience of working with pen and notebooks is extended to create a three dimensional two-handed interface, that supports easy-to-understand manipulation tasks in augmented and virtual environments. In the design step we take advantage from the freedom, given by our very low demands on hardware and augment form and functionality to this device. On the basis of examples from object manipulation, augmented research environments and scientific visualization we show the generality of applicability. Although being in the first stages implementation, we consider the wide spectrum of suitability for different purposes.

In current interfaces, users select objects, apply operations, and change viewing parameters in distinct steps that require switching attention among several screen areas. Our See-Through Interface ™ software reduces steps by locating tools on a transparent sheet that can be moved over applications with one hand using a trackball, while the other hand controls a mouse cursor. The user clicks through a tool onto application objects, simultaneously selecting an operation and an operand. Tools may include graphical filters that display a customized view of application objects. Compared to traditional interactors, these tools save steps, require no permanent screen space, reduce temporal modes, apply to multiple applications, and facilitate customization. This paper presents a taxonomy of see-through tools that considers variations in each of the steps they perform. As examples, we describe particular see-through tools that perform graphical editing and text editing operations.