This paper reviews a number of recently available techniques in contentanalysis of visual media and their application to the indexing, retrieval,abstracting, relevance assessment, interactive perception, annotation and re-use of visualdocuments. 1. Background A few years ago, the problems of representation and retrieval of visualmedia were confined to specialized image databases (geographical, medical, pilot experimentsin computerized slide libraries), in the professional applications of the audiovisualindustries (production, broadcasting and archives), and in computerized training or education. The presentdevelopment of multimedia technology and information highways has put content processing of visualmedia at the core of key application domains: digital and interactive video, large distributed digital libraries, multimedia publishing. Though the most important investments have been targeted at the information infrastructure (networks, servers, coding and compression, deliverymodels, multimedia systems architecture), a growing number of researchers have realized thatcontent processing will be a key asset in putting together successful applications. The need for contentprocessing techniques has been made evident from a variety of angles, ranging from achievingbetter quality in compression, allowing user choice of programs in video-on-demand, achieving betterproductivity in video production, providing access to large still image databases or integrating still images and video in multimedia publishing and cooperative work. Content-based retrieval of visual media and representation of visualdocuments in human-computer interfaces are based on the availability of content representationdata (time-structure for

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.

This paper reports two studies that measured the effects of different “video skim ” techniques on comprehension, navigation, and user satisfaction. Video skims are compact, content-rich abstractions of longer videos, condensations that preserve frame rate while greatly reducing viewing time. Their characteristics depend on the image- and audio-processing techniques used to create them. Results from the initial study helped refine video skims, which were then reassessed in the second experiment. Significant benefits were found for skims built from audio sequences meeting certain criteria.

Storage and retrieval of multimedia has become a requirement for many contemporary information systems. These systems need to provide browsing, querying, navigation and sometimes composition capabilities involving various forms of media. In this survey we review techniques and systems for image and video retrieval. We first look at visual and non-visual features for image retrieval and techniques for using them. Temporal aspects of video retrieval are discussed next. We review several research and commercial systems including WWW-based systems and conclude with future directions. 1 Introduction The increasing availability of multimedia information combined with the decreasing storage and processing costs have changed the requirements on information systems drastically. Today, general purpose database systems are incorporating support for multimedia storage and retrieval, as well as features which used to be found in specialized imaging systems or multimedia databases. Increased use of...

Abstract—This paper presents the semantic pathfinder architecture for generic indexing of multimedia archives. The semantic pathfinder extracts semantic concepts from video by exploring different paths through three consecutive analysis steps, which we derive from the observation that produced video is the result of an authoring-driven process. We exploit this authoring metaphor for machine-driven understanding. The pathfinder starts with the content analysis step. In this analysis step, we follow a data-driven approach of indexing semantics. The style analysis step is the second analysis step. Here, we tackle the indexing problem by viewing a video from the perspective of production. Finally, in the context analysis step, we view semantics in context. The virtue of the semantic pathfinder is its ability to learn the best path of analysis steps on a per-concept basis. To show the generality of this novel indexing approach, we develop detectors for a lexicon of 32 concepts and we evaluate the semantic pathfinder against the 2004 NIST TRECVID video retrieval benchmark, using a news archive of 64 hours. Top ranking performance in the semantic concept detection task indicates the merit of the semantic pathfinder for generic indexing of multimedia archives. Index Terms—Video analysis, concept learning, benchmarking, content analysis and indexing, multimedia information systems, pattern recognition. 1

As digital video becomes more pervasive, e#cient ways of searching and annotating video according to content will be increasingly important. Such tasks arise, for example, in the management of digital video libraries for content-based retrieval and browsing. In this paper, we develop tools based on camera motion for analyzing and annotating a class of structured video using the low-level information available directly from MPEG compressed video. In particular, we show that in certain structured settings it is possible to obtain reliable estimates of camera motion by directly processing data easily obtained from the MPEG format. Working directly with the compressed video greatly reduces the processing time and enhances storage e#ciency.

Filmstrips and video skims are two presentation schemes for abstracting information in a digital video segment. Filmstrips present information all at once in a static form, while video skims are played and disclose information temporally. This paper discusses the evolution of the filmstrip and skim interfaces in the Informedia Digital Video Library. Filmstrips are commonly deployed as interfaces for video and image libraries, but we found initial Informedia filmstrips and skims received little use. We discuss the interface considerations motivating the redesign of filmstrips and skims to adjust their presentations dynamically based on user context and preference.

Extracting information from the ever growing stream of multimedia data is becoming increasingly difficult. One of the main reasons lies within the unstructured way multimedia data are usually presented. Audio-visual material represents a large part of current multimedia material and can be structured in meaningful ways due to the nature of visual communication. This paper surveys several approaches and algorithms that have been recently developed to help in automatically structuring audio-visual data, both for annotation and access

Multimedia abstractions form essential components of digital video libraries because they enable a user to determine a video’s distinguishing content without investing long viewing times or requiring high networktransfer speeds. This paper presents usage and evaluation data for abstractions implemented the Informedia Digital Video Library, and discusses implications for video delivery over the Web.

Multimedia information retrieval is a challenging problem because multimedia information is not inherently structured. Jabber is an experimental system that attempts to bring some structure to this task. Jabber allows users to retrieve records of videoconferences based upon the concepts discussed. In this paper we introduce ConceptFinder, a sub-system within Jabber, and show how it is able to process the spoken text of a meeting into meeting topics. ConceptFinder can make subtle distinctions among different senses of the same words, and is able to summarize a set of related words, giving a name to each topic. Users can then use this name to query or browse the stored multimedia, through Jabber's user interface. By presenting information that closely matches a user's expectations, the challenge of multimedia retrieval is rendered more tractable. Keywords Multimedia indexing, information retrieval and browsing, concept clustering INTRODUCTION Multimedia holds tremendous appeal becau...