The goal of the research described in this paper is to develop an application-independent presentation tool that automatically designs effective graphical presentations (such as bar charts, scatter plots, and connected graphs) of relational information. Two problems are raised by this goal: The codifi-cation of graphic design criteria in a form that can be used by the presentation tool, and the generation of a wide variety of designs so that the presentation tool can accommodate a wide variety of information. The approach described in this paper is based on the view that graphical presentations are sentences of graphical languages. The graphic design issues are codified as expressiveness and effectiveness criteria for graphical languages. Expressiveness criteria determine whether a graphical language can express the desired information. Effectiveness criteria determine whether a graphical language exploits the capabilities of the output medium and the human visual system. A wide variety of designs can be systematically generated by using a composition algebra that composes a small set of primitive graphical languages. Artificial intelligence techniques are used to implement a prototype presentation tool called APT (A Presentation Tool), which is based on the composition algebra and the graphic design criteria.

Advances in information technologies and telecommunications are making available a very large number of possible input/output modality combinations which are potentially useful for the expression and exchange of information in human-human-system interaction (HHSI). However, standard or shared foundations consisting of all available and possible elementary, or unimodal, input and output modalities is still lacking. A common representation format for describing the relevant properties of individual unimodal modalities is lacking as well. Ideally, such foundations could enable designers of multimodal interaction to select an appropriate set of modalitie s for an application to be designed, once its information representation and exchange requirements have been specified. This chapter proposes Modality Theory-based solutions and discusses alternative approaches to major parts of this practical problem of modality selection. The solutions cover the generation, at descending levels of abstraction, of taxonomies of unimodal input and output modal ities from basic properties in the media of graphics, acoustics and haptics. The result is a practical `designer's t oolbox' of unimodal modalities. A first common representation format for unimodal modalities is presented. The taxonomy and the unimodal modalities as represented in the common format are then brought to the test by addressing the complex issue of when (not) to use speech in practical applications. Finally, empirical and theoretical approaches to the combinatorial explosion of modality combinations in multimodal systems are discussed.

The paper addresses an issue that must be resolved in order to produce a scientifically sound and practically useful reference model for intelligent multimedia presentation systems (IMP systems), namely that of providing a systematic understanding of the types of output information to be presented by IMP systems. The term `medium', though well-defined, is too coarse-grained for distinguishing between different types of output information. The paper introduces the notion of (representational) `modalities' to enable sufficiently fine-grained distinctions to be made. For the term itself to be meaningful, `multimodal' presentations must be composed of unimodal representations. In the approach presented, unimodal representations are defined from a small number of basic properties whose combinations specify the `generic' level of a taxonomy of unimodal output modalities. To be scientifically sound as well as practically useful, the taxonomy must satisfy requirements of completeness, orthogonality, relevance and intuitiveness. The generic level of the taxonomy turns out to be too abstract to satisfy these requirements. By consequence, an `atomic' and a `sub-atomic' level are generated by analysis from the generic level, which satisfy the mentioned requirements. Based on the atomic and sub-atomic levels, all possible multimodal representations in the media of graphics, acoustics and haptics can now be generated by composition. The concluding discussion raises the issues of empirical validation of the taxonomy, its practical usefulness, and of expanding the approach to cover input modalities of information as well as user-system interactivity.