Abstract. The PASCAL Visual Object Classes Challenge ran from February to March 2005. The goal of the challenge was to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). Four object classes were selected: motorbikes, bicycles, cars and people. Twelve teams entered the challenge. In this chapter we provide details of the datasets, algorithms used by the teams, evaluation criteria, and results achieved. 1

We have witnessed great interest and a wealth of promise in content-based image retrieval as an emerging technology. While the last decade laid foundation to such promise, it also paved the way for a large number of new techniques and systems, got many new people involved, and triggered stronger association of weakly related fields. In this article, we survey almost 300 key theoretical and empirical contributions in the current decade related to image retrieval and automatic image annotation, and in the process discuss the spawning of related subfields. We also discuss significant challenges involved in the adaptation of existing image retrieval techniques to build systems that can be useful in the real world. In retrospect of what has been achieved so far, we also conjecture what the future may hold for image retrieval research.

The last decade has witnessed great interest in research on content-based image retrieval. This has paved the way for a large number of new techniques and systems, and a growing interest in associated fields to support such systems. Likewise, digital imagery has expanded its horizon in many directions, resulting in an explosion in the volume of image data required to be organized. In this paper, we discuss some of the key contributions in the current decade related to image retrieval and automated image annotation, spanning 120 references. We also discuss some of the key challenges involved in the adaptation of existing image retrieval techniques to build useful systems that can handle real-world data. We conclude with a study on the trends in volume and impact of publications in the field with respect to venues/journals and sub-topics.

Abstract. Automatic image annotation has attracted a lot of attention recently as a method for facilitating semantic indexing and text-based retrieval of visual content. In this paper, we propose the use of multiple Self-Organizing Maps in modeling various semantic concepts and annotating new input images automatically. The effect of the semantic gap is compensated by annotating multiple images concurrently, thus enabling more accurate estimation of the semantic concepts ’ distributions. The presented method is applied to annotating images from a freely-available database consisting of images of different semantic categories. 1

Abstract. Content-based image retrieval (CBIR) is an emerging research field, studying retrieval of images from unannotated databases. In CBIR, images are indexed on the basis of low-level statistical features that can be automatically derived from the images. Due to the gap between high-level semantic concepts and low-level visual features, the performance of CBIR applications often remains quite modest. One method for improving CBIR results is to try to learn the user’s preferences with intra-query learning methods such as relevance feedback. However, relevance feedback provides user interaction information which can automatically be used also in long-term or inter-query learning. In this paper, a method for using long-term learning in our PicSOM system is presented. The performed experiments show that the system readily supports the presented user interaction feature and that the efficiency of the system can be substantially increased by using it in parallel with the MPEG-7 visual descriptors.

Abstract. In content-based image retrieval (CBIR), the images in a database are indexed on the basis of low-level statistical features that can be automatically derived from the images. Due to the semantic gap, the performance of CBIR systems often remains quite modest especially on broad image domains. One method for improving the results is to incorporate automatic image classification methods to the CBIR system. The resulting subsets can be indexed separately with features suitable for those particular images or used to limit an image query only to certain promising image subsets. In this paper, a method for supporting different types of image subsets within a generic framework based on multiple parallel Self-Organizing Maps and binary clusterings is presented. 1

Our experiments in TRECVID 2005 include participation in the high-level feature extraction and search tasks. In the highlevel feature extraction task, we applied a method of representing semantic concepts as class models on a set of parallel Self-Organizing Maps (SOMs). We submitted one run, A PicSOM 1, in which we applied a feature selection scheme for each concept separately. The results showed that the SOM-based class models can be used for representing semantic concepts on multimodal feature indices and that the proposed method is suitable for detecting video shots with specific semantic content. In the search task, we submitted a total of seven runs (three automatic, three manual, and one interactive run). Our main motivation was to study the utilization of parallel multimodal features and class models compared to using only text-based queries. The overall settings for the runs were as follows: • F A 1 SOM-F1 7: a baseline automatic run using only ASR/MT output • F A 2 SOM-F2 3: an automatic run using ASR/MT output, multimodal features, and class models • F A 2 SOM-F3 5: an automatic run using multimodal features and class models • M A 1 SOM-M1 6: a baseline manual run using only ASR/MT output • M A 2 SOM-M2 4: a manual run using ASR/MT output and multimodal features • M A 2 SOM-M3 2: a manual run using ASR/MT output, multimodal features, and class models • I A 2 SOM-I 1: an interactive run Both in the automatic and manual experiments, we observed that the proposed method is able to combine the text query, multimodal features and class models successfully. In both cases, the overall best results are obtained using all three information sources with the MAP value being nearly double when compared to text-only search. Our small-scale interactive search experiments were performed with our prototype retrieval interface supporting only relevance feedback-based retrieval. Still, the experiments demonstrate that the proposed method can also be used in an interactive setting, where the search is guided with iterative feedback from the user.

Our experiments in TRECVID 2006 include participation in the shot boundary detection, high-level feature extraction, and search tasks, using a common system framework based on multiple parallel Self-Organizing Maps (SOMs). In the shot boundary detection task we projected feature vectors calculated from successive frames on parallel SOMs and monitored the trajectories to detect the shot boundaries. We submitted the following ten runs: • PicSOM CA: cut-optimized using all the training videos • PicSOM GA: gradual-optimized using all the training videos • PicSOM BA: optimized for both cuts and gradual transitions using all the training videos • PicSOM CN: cut-optimized using only the news videos (without the NASA videos) • PicSOM GN: gradual-optimized using only the news videos • PicSOM CS: cut-optimized using channel-specific training videos • PicSOM GS: gradual-optimized using channel-specific training videos • PicSOM CNF: cut-optimized using only the news videos and only a few features • PicSOM CNE: cut-optimized using only the news videos and one additional edge feature • PicSOM CAE: cut-optimized using all the training videos and one additional edge feature

The amount of available digital multimedia has seen exponential growth in recent years. While advances have been made in the indexing and searching of images and videos, less focus has been given to aiding users in the interactive exploration of large datasets. In this paper a new framework, called visual islands, is proposed that reorganizes image query results from an initial search or even a general photo collection using a fast, non-global feature projection to compute 2D display coordinates. A prototype system is implemented and evaluated with three core goals: fast browsing, intuitive display, and non-linear exploration. Using the TRECVID2005[15] dataset, 10 users evaluated the goals over 24 topics. Experiments show that users experience improved comprehensibility and achieve a significant page-level precision improvement with the visual islands framework over traditional paged browsing.

Abstract. The content-based image retrieval (CBIR) system PicSOM uses a variety of low-level visual features as an indexing mechanism for an image database. In this paper we describe the implementation of segmentation into the PicSOM framework. That is, we have modified the system to use image segments as a supplement to entire images in order to improve the retrieval accuracy. In a series of experiments, we compare this new method to the baseline PicSOM system. The results confirm that using both segments and entire images together always increases the precision of retrieval. 1