We describe an inertial gesture recognition framework composed of three parts. The first is a compact, wireless six-axis inertial measurement unit to fully capture three-dimensional motion. The second, a gesture recognition algorithm, analyzes the data and categorizes it on an axis-by-axis basis as simple motions (straight line, twist, etc.) with magnitude and duration. The third allows an application designer to combine recognized gestures both concurrently and consecutively to create specific composite gestures can then be set to trigger output routines.

Inertial measurement components, which sense either acceleration or angular rate, are being embedded into common user interface devices more frequently as their cost continues to drop dramatically. These devices hold a number of advantages over other sensing technologies: they measure relevant parameters for human interfaces and can easily be embedded into wireless, mobile platforms. The work in this dissertation demonstrates that inertial measurement can be used to acquire rich data about human gestures, that we can derive efficient algorithms for using this data in gesture recognition, and that the concept of a parameterized atomic gesture recognition has merit. Further we show that a framework combining these three levels of description can be easily used by designers to create robust applications.

This paper considers the control of computer games through players interacting with the physical environment around them in a natural and appropriate manner. This is achieved using a sentient computing system. Such a system senses the location, motions, actions and even physiological responses of users. This sensory data can be used to interface and control the game. A good example is 3D rst-person games, and we demonstrate a system in which actions in the game are mapped to similar actions in the real world. The aim is to give a more involving experience through the unencumbered ubiquitous control of the game. 1

For molecular modeling, chemical structures have to be understood and imagined both in their three-dimensional spatial extent and in their dynamic behavior. We have developed an AR-based system for tangible interaction with molecules using optical markers. When users bring several molecules close to one another, potential bonds are shown and the molecules dynamically change their 3D structure according to potential chemical reactions. A problem arises when users also need to select one such bond from of a multitude of potential bonds while already using both hands to manipulate the molecules. We present two gesture-based techniques, shake-based and proximity-based to solve this problem. We report on user tests evaluating these techniques with respect to speed, precision and user acceptance.

Mobile and wearable computers will increasingly be able to deliver interactive 3D graphical environments. In the near future, 3D terrain visualizations will be an important class of applications for users of mobile and wearable computers. These visualizations can be used to present additional information about a user's surrounding environment and thus enhance the user's environmental awareness and understanding. Such visualization will be useful in problem domains that require geospatial knowledge and coordination of tasks over large geographic areas. Recent advances in mobile computing hardware, sensing, and networking make such visualization applications feasible. However, a number of areas require further research. In this paper, seven key challenges to designing and building such systems are discussed. This paper seeks to increase interest in such applications, create a dialog centered on those key issues, and engage researchers from the virtual reality field and other communities.

The Itsy pocket computer is a powerful information device small enough to be comfortably worn or carried. It was created to support research in user interfaces and applications. A 32-bit, 200MHz microprocessor and 32 MB each of flash memory and DRAM made it the first small handheld able to run traditional desktop applications such as continuous speech recognition, speech synthesis, multimedia, and video games. Itsy offers fine-grain control of the hardware to support flexible power management and monitoring. Comprehensive expansion capability via daughtercards makes it easy to extend the system. The Linux operating system, with extensions for a flash file system, resource sharing, and power management, eased porting of several advanced application environments (XWindows, Java and Squeak) and many applications. Our user interface experiments confirm that traditional desktop methods do not extend well to interactive applications on small devices, but that gesture- and speech-based approaches have great potential. Our power measurements and analysis show that 300 -- 900 mW, which our battery can easily supply, is sufficient to run a variety of the most demanding applications.