Abstract—This paper presents the semantic pathfinder architecture for generic indexing of multimedia archives. The semantic pathfinder extracts semantic concepts from video by exploring different paths through three consecutive analysis steps, which we derive from the observation that produced video is the result of an authoring-driven process. We exploit this authoring metaphor for machine-driven understanding. The pathfinder starts with the content analysis step. In this analysis step, we follow a data-driven approach of indexing semantics. The style analysis step is the second analysis step. Here, we tackle the indexing problem by viewing a video from the perspective of production. Finally, in the context analysis step, we view semantics in context. The virtue of the semantic pathfinder is its ability to learn the best path of analysis steps on a per-concept basis. To show the generality of this novel indexing approach, we develop detectors for a lexicon of 32 concepts and we evaluate the semantic pathfinder against the 2004 NIST TRECVID video retrieval benchmark, using a news archive of 64 hours. Top ranking performance in the semantic concept detection task indicates the merit of the semantic pathfinder for generic indexing of multimedia archives. Index Terms—Video analysis, concept learning, benchmarking, content analysis and indexing, multimedia information systems, pattern recognition. 1

Abstract — In this paper, we propose an automatic video retrieval method based on high-level concept detectors. Research in video analysis has reached the point where over 100 concept detectors can be learned in a generic fashion, albeit with mixed performance. Such a set of detectors is very small still compared to ontologies aiming to capture the full vocabulary a user has. We aim to throw a bridge between the two fields by building a multimedia thesaurus, i.e. a set of machine learned concept detectors that is enriched with semantic descriptions and semantic structure obtained from WordNet. Given a multimodal user query, we identify three strategies to select a relevant detector from this thesaurus, namely: text matching, ontology querying, and semantic visual querying. We evaluate the methods against the automatic search task of the TRECVID 2005 video retrieval benchmark, using a news video archive of 85 hours in combination with a thesaurus of 363 machine learned concept detectors. We assess the influence of thesaurus size on video search performance, evaluate and compare the multimodal selection strategies for concept detectors, and finally discuss their combined potential using oracle fusion. The set of queries in the TRECVID 2005 corpus is too small to be definite in our conclusions, but the results suggest promising new lines of research. Index Terms — Video retrieval, concept learning, knowledge modeling, content analysis and indexing, multimedia information systems I.

Abstract—Effective video retrieval is the result of an interplay between interactive query selection, advanced visualization of results, and a goal-oriented human user. Traditional interactive video retrieval approaches emphasize paradigms, such as query-by-keyword and query-by-example, to aid the user in the search for relevant footage. However, recent results in automatic indexing indicate that query-by-concept is becoming a viable resource for interactive retrieval also. We propose in this paper a new video retrieval paradigm. The core of the paradigm is formed by first detecting a large lexicon of semantic concepts. From there, we combine query-by-concept, query-by-example, query-by-keyword, and user interaction into the MediaMill semantic video search engine. To measure the impact of increasing lexicon size on interactive video retrieval performance, we performed two experiments against the 2004 and 2005 NIST TRECVID benchmarks, using lexicons containing 32 and 101 concepts, respectively. The results suggest that from all factors that play a role in interactive retrieval, a large lexicon of semantic concepts matters most. Indeed, by exploiting large lexicons, many video search questions are solvable without using query-by-keyword and query-by-example. In addition, we show that the lexicon-driven search engine outperforms all state-of-the-art video retrieval systems in both TRECVID 2004 and 2005. Index Terms—Benchmarking, concept learning, content analysis and indexing, interactive systems, multimedia information systems, video retrieval. I.

This year the UvA-MediaMill team participated in the Feature Extraction and Search Task. We developed a generic approach for semantic concept classification using the semantic value chain. The semantic value chain extracts concepts from video documents based on three consecutive analysis links, named the content link, the style link, and the context link. Various experiments within the analysis links were performed, showing amongst others the merit of processing beyond key frames, the value of style elements, and the importance of learning semantic context. For all experiments a lexicon of 32 concepts was exploited, 10 of which are part of the Feature Extraction Task. Top three system-based ranking in 8 out of the 10 benchmark concepts indicates that our approach is very promising. Apart from this, the lexicon of 32 concepts proved very useful in an interactive search scenario with our semantic video search engine, where we obtained the highest mean average precision of all participants. 1

This study surveys current trends/methods in video retrieval. The major themes covered by the study include shot segmentation, key frame extraction, feature extraction, clustering, indexing and video retrieval-by similarity, probabilistic, transformational, refinement and relevance feedback. This work has done in an aim to assist the upcoming researchers in the field of video retrieval, to know about the techniques and methods available for video retrieval.

Abstract. In this paper we describe a Semantic Grid application designed to enable museums and indigenous communities in distributed locations, to collaboratively discuss, describe and annotate digital objects and documents in museums that originally belonged to or are of cultural or historical significance to indigenous groups. By extending and refining an existing application, Vannotea, we enable users on access grid nodes to collaboratively attach descriptive, rights and tribal care metadata and annotations to digital images, video or 3D representations. The aim is to deploy the software within museums to enable the traditional owners to describe and contextualize museum content in their own words and from their own perspectives. This sharing and exchange of knowledge will hopefully revitalize cultures eroded through colonization and globalization and repair and strengthen relationships between museums and indigenous communities. 1

methods, mpeg-7

Digital video now plays an important role in medical education and healthcare, but our ability to automatic video indexing at semantic level is currently primitive. In this paper, we propose a novel framework to enable more effective semantic video classification and indexing in a specific surgery education video domain. Specifically, this framework includes: (a) A novel semantic-sensitive video content characterization and representation framework by using principal video shots and their perceptual multimodal features. (b) A novel semantic medical concept interpretation technique by using flexible mixture model. (c) A semantic video classifier by using an adaptive Expectation-Maximization (EM) algorithm for automatic parameter estimation and model selection (i.e., selecting the optimal number of mixture Gaussian components). Since more effective video content characterization framework has been integrated with an adaptive EM algorithm for video classification, our semantic video classifier has improved the classification accuracy significantly. For skin classification, its accuracy is close to 95.5%. For semantic surgical video classification, it achieves overall ≈ 84.6 % accuracy.

The dream of pervasive multimedia retrieval and reuse will not be realized without incorporating semantics in the multimedia database. As video data is penetrating many information systems, the need for database support for video data evolves. Hence, we propose an innovative database modeling mechanism called Hierarchical Markov Model Mediator (HMMM) which integrates lowlevel features, semantic concepts, and high-level user perceptions for modeling and indexing multiple-level video objects to facilitate temporal pattern retrieval. Different from the existing database modeling methods, our approach carries a stochastic and dynamic process in both search and similarity calculation. In the retrieval of semantic event patterns, HMMM always tries to traverse the right path and therefore it can assist in retrieving more accurate patterns quickly with lower computational costs. Moreover, HMMM supports feedbacks and learning strategies, which can proficiently assure the continuous improvements of the overall performance. 1.

To support more effective video retrieval at semantic level, we introduce a novel framework to achieve semantic video classification. This novel framework includes: (a) A semantic-senstive video content representation framework via principal video shots to enhance the quality of features (i.e., the ability of the selected low-level multimodal perceptual features to discriminate among various semantic video concepts); (b) A semantic video concept interpretation framework via flexible mixture model to bridge the semantic gap between the semantic video concepts and the low-level multimodal perceptual features; (c) A novel concept learning technique to integrate unlabeled samples with labeled samples for more accurate classifier training. Experimental results on semantic medical video classification are also presented to evaluate the performance of the proposed framework.