We present SketchREAD, a multi-domain sketch recognition engine capable of recognizing freely hand-drawn diagrammatic sketches. Current computer sketch recognition systems are difficult to construct, and either are fragile or accomplish robustness by severely limiting the designer’s drawing freedom. Our system can be applied to a variety of domains by providing structural descriptions of the shapes in that domain; no training data or programming is necessary. Robustness to the ambiguity and uncertainty inherent in complex, freely-drawn sketches is achieved through the use of context. The system uses context to guide the search for possible interpretations and uses a novel form of dynamically constructed Bayesian networks to evaluate these interpretations. This process allows the system to recover from low-level recognition errors (e.g., a line misclassified as an arc) that would otherwise result in domain level recognition errors. We evaluated SketchREAD on real sketches in two domains— family trees and circuit diagrams—and found that in both domains the use of context to reclassify low-level shapes significantly reduced recognition error over a baseline system that did not reinterpret low-level classifications. We also discuss the system’s potential role in sketch-based user interfaces.

Zhai and Kristensson (2003) presented a method of speedwriting for pen-based computing which utilizes gesturing on a stylus keyboard for familiar words and tapping for others. In SHARK 2, we eliminated the necessity to alternate between the two modes of writing, allowing any word in a large vocabulary (e.g. 10,000-20,000 words) to be entered as a shorthand gesture. This new paradigm supports a gradual and seamless transition from visually guided tracing to recall-based gesturing. Based on the use characteristics and human performance observations, we designed and implemented the architecture, algorithms and interfaces of a high-capacity multi-channel pen-gesture recognition system. The system’s key components and performance are also reported.

Abstract. In this paper, we argue that current user interface modeling tools are developed using a formalism-centric approach that does not support the needs of modern software development. In order to solve this problem we need both usable and expressive notations and tools that enable the creation of userinterface specifications that leverage the design and thought process. In this paper we present the CanonSketch tool. CanonSketch supports a new UI specification language – Canonical Abstract Prototypes (CAP) – that bridges the gap between envisioned user behavior and the concrete user interface. The tool also supports two additional and synchronized views of the UI: the Wisdom UML presentation extension and concrete HTML user interfaces. In this way the tool seamlessly supports designers while switching from high level abstract views of the UI and low-level concrete realizations. 1

research.microsoft.com

Abstract not found

Multi-fidelity prototyping combines within a single user interface prototype several elements whose representations are reproduced with different levels of fidelity with respect to the final user interface: no fidelity at all, low fidelity, medium fidelity, and high fidelity. In order to effectively and efficiently support multi-fidelity, an original set of techniques is defined and discussed: multiple representation manipulation by sketching and gesture recognition, smooth transition between any representation at any time, prototype reusability, multi-platform support, and multiple user interface prototyping facilities. The full implementation of these techniques in prototyping software provides designers and developers with a unique environment for exploring multiple designs with unprecedented support for quickly designing interfaces from scratch or from previously existing design templates. An experimental study reveals that the multiple representation manipulation together with smooth transition represents a valuable advantage for naturally designing user interfaces. The prototyping software supports several aspects involved in the user interface development life cycle and is convenient for non-WIMP user interfaces.

Abstract. Sketching activities are widely adopted during early design phases of user interface development to convey informal specifications of the interface presentation and dialog. Designers or even end users can sketch some or all of the future interface they want. With the ever increasing availability of different computing platforms, a need arises to continuously support sketching across these platforms with their various programming languages, interface development environments and operating systems. To address needs along these dimensions, which pose new challenges to user interface sketching tools, SketchiXML is a multi-platform multi-agent interactive application that enable designers and end users to sketch user interfaces with different levels of details and support for different contexts of use. The results of the sketching are then analyzed to produce interface specifications independently of any context, including user and platform. These specifications are exploited to progressively produce one or several interfaces, for one or many users, platforms, and environments. 1

Designing UIs that run across multiple devices is increasingly important. To address this, we have created a prototyping tool called Damask, which targets web UIs that run on PCs and mobile phones, and prompt-and-response style voice UIs. In Damask, designers sketch out their design for one device while using design patterns to specify higher-level concepts within their design. Damask’s patterns include pre-built UI fragments that are already optimized for each device. Designers also use layers to specify which UI parts are common across devices and which are specific to one device. Damask uses the sketches and patterns to generate designs for the other devices, which the designers can refine. A study performed with 12 professional UI designers found that, in the early stages, designers using patterns and layers in Damask created cross-device UIs that are rated at least as good as those created without patterns and layers, without more time. ACM Classification: D.2.2 [Software Engineering]: Design

SketchWizard allows designers to create Wizard of Oz prototypes of pen-based user interfaces in the early stages of design. In the past, designers have been inhibited from participating in the design of pen-based interfaces because of the inadequacy of paper prototypes and the difficulty of developing functional prototypes. In SketchWizard, designers and end users share a drawing canvas between two computers, allowing the designer to simulate the behavior of recognition or other technologies. Special editing features are provided to help designers respond quickly to end-user input. This paper describes the SketchWizard system and presents two evaluations of our approach. The first is an early feasibility study in which Wizard of Oz was used to prototype a pen-based user interface. The second is a laboratory study in which designers used SketchWizard to simulate existing pen-based interfaces. Both showed that end users gave valuable feedback in spite of delays between end-user actions and wizard updates.

Because most animation tools are complex and timeconsuming to learn and use, most animations today are created by experts. To help novices create a wide range of animations quickly, we have developed a general-purpose, informal, 2D animation sketching system called K-Sketch. Field studies investigating the needs of animators and would-be animators helped us collect a library of usage scenarios for our tool. A novel optimization technique enabled us to design an interface that is simultaneously fast, simple, and powerful. The result is a pen-based system that relies on users ’ intuitive sense of space and time while still supporting a wide range of uses. In a laboratory experiment that compared K-Sketch to a more formal animation tool (PowerPoint), participants worked three times faster, needed half the learning time, and had significantly lower cognitive load with K-Sketch.