A Survey on Multimodal Video Representation for Semantic Retrieval
---
paper_title: Multimodal Video Indexing: A Review of the State-of-the-art

paper_content:

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.
---
paper_title: The language of new media

paper_content:

"In this book Lev Manovich offers the first systematic and rigorous theory of new media. He places new media within the histories of visual and media cultures of the last few centuries. He discusses new media's reliance on conventions of old media, such as the rectangular frame and mobile camera, and shows how new media works create the illusion of reality, address the viewer, and represent space. He also analyzes categories and forms unique to new media, such as interface and database. ::: ::: Manovich uses concepts from film theory, art history, literary theory, and computer science and also develops new theoretical constructs, such as cultural interface, spatial montage, and cinegratography. The theory and history of cinema play a particularly important role in the book. Among other topics, Manovich discusses parallels between the histories of cinema and of new media, digital cinema, screen and montage in cinema and in new media, and historical ties between avant-garde film and new media." -- Publisher's website.
---
paper_title: Media streams: representing video for retrieval and repurposing

paper_content:

Thesis (Ph. D.)--Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1995.
---
paper_title: Multimedia content processing through cross-modal association

paper_content:

Multimodal information processing has received considerable attention in recent years. The focus of existing research in this area has been predominantly on the use of fusion technology. In this paper, we suggest that cross-modal association can provide a new set of powerful solutions in this area. We investigate different cross-modal association methods using the linear correlation model. We also introduce a novel method for cross-modal association called Cross-modal Factor Analysis (CFA). Our earlier work on Latent Semantic Indexing (LSI) is extended for applications that use off-line supervised training. As a promising research direction and practical application of cross-modal association, cross-modal information retrieval where queries from one modality are used to search for content in another modality using low-level features is then discussed in detail. Different association methods are tested and compared using the proposed cross-modal retrieval system. All these methods achieve significant dimensionality reduction. Among them CFA gives the best retrieval performance. Finally, this paper addresses the use of cross-modal association to detect talking heads. The CFA method achieves 91.1% detection accuracy, while LSI and Canonical Correlation Analysis (CCA) achieve 66.1% and 73.9% accuracy, respectively. As shown by experiments, cross-modal association provides many useful benefits, such as robust noise resistance and effective feature selection. Compared to CCA and LSI, the proposed CFA shows several advantages in analysis performance and feature usage. Its capability in feature selection and noise resistance also makes CFA a promising tool for many multimedia analysis applications.
---
paper_title: Matching Words and Pictures

paper_content:

We present a new approach for modeling multi-modal data sets, focusing on the specific case of segmented images with associated text. Learning the joint distribution of image regions and words has many applications. We consider in detail predicting words associated with whole images (auto-annotation) and corresponding to particular image regions (region naming). Auto-annotation might help organize and access large collections of images. Region naming is a model of object recognition as a process of translating image regions to words, much as one might translate from one language to another. Learning the relationships between image regions and semantic correlates (words) is an interesting example of multi-modal data mining, particularly because it is typically hard to apply data mining techniques to collections of images. We develop a number of models for the joint distribution of image regions and words, including several which explicitly learn the correspondence between regions and words. We study multi-modal and correspondence extensions to Hofmann's hierarchical clustering/aspect model, a translation model adapted from statistical machine translation (Brown et al.), and a multi-modal extension to mixture of latent Dirichlet allocation (MoM-LDA). All models are assessed using a large collection of annotated images of real scenes. We study in depth the difficult problem of measuring performance. For the annotation task, we look at prediction performance on held out data. We present three alternative measures, oriented toward different types of task. Measuring the performance of correspondence methods is harder, because one must determine whether a word has been placed on the right region of an image. We can use annotation performance as a proxy measure, but accurate measurement requires hand labeled data, and thus must occur on a smaller scale. We show results using both an annotation proxy, and manually labeled data.
---
paper_title: Highlights modeling and detection in sports videos

paper_content:

Automatic annotation of semantic events allows effective retrieval of video content. In this work, we present solutions for highlights detection in sports videos. This application is particularly interesting for broadcasters, since they extensively use manual annotation to select interesting highlights that are edited to create new programmes. The proposed approach exploits the typical structure of a wide class of sports videos, namely, those related to sports which are played in delimited venues with playfields of well known geometry, like soccer, basketball, swimming, track and field disciplines, and so on. For this class of sports, a modeling scheme based on a limited set of visual cues and on finite state machines (FSM) that encode the temporal evolution of highlights is presented. Algorithms for model checking and for visual cues estimation are discussed, as well as applications of the representation to different sport domains.
---
paper_title: Multimedia content analysis-using both audio and visual clues

paper_content:

Multimedia content analysis refers to the computerized understanding of the semantic meanings of a multimedia document, such as a video sequence with an accompanying audio track. With a multimedia document, its semantics are embedded in multiple forms that are usually complimentary of each other, Therefore, it is necessary to analyze all types of data: image frames, sound tracks, texts that can be extracted from image frames, and spoken words that can be deciphered from the audio track. This usually involves segmenting the document into semantically meaningful units, classifying each unit into a predefined scene type, and indexing and summarizing the document for efficient retrieval and browsing. We review advances in using audio and visual information jointly for accomplishing the above tasks. We describe audio and visual features that can effectively characterize scene content, present selected algorithms for segmentation and classification, and review some testbed systems for video archiving and retrieval. We also describe audio and visual descriptors and description schemes that are being considered by the MPEG-7 standard for multimedia content description.
---
paper_title: Cinematic primitives for multimedia

paper_content:

The development of robust frameworks in interactive multimedia for representing story elements to the machine so that they can be retrieved in multiple contexts is addressed. Interactive multimedia is discussed as a user-directed form of storytelling, and the nature of cinematic storytelling is examined. It is proposed that content can be represented in layers. This model for layered information will allow programs to take advantage of the relation between cinematic sequences and the world they represent. The collection of content by the camera and microphone is considered in this context. The use of the methodology to build meaningful, context-rich sequences is discussed. >
---
paper_title: Multimodal Video Indexing: A Review of the State-of-the-art

paper_content:

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.
---
paper_title: A design space for multimodal systems: concurrent processing and data fusion

paper_content:

Multimodal interaction enables the user to employ different modalities such as voice, gesture and typing for communicating with a computer. This paper presents an analysis of the integration of multiple communication modalities within an interactive system. To do so, a software engineering perspective is adopted. First, the notion of “multimodal system” is clarified. We aim at proving that two main features of a multimodal system are the concurrency of processing and the fusion of input/output data. On the basis of these two features, we then propose a design space and a method for classifying multimodal systems. In the last section, we present a software architecture model of multimodal systems which supports these two salient properties: concurrency of processing and data fusion. Two multimodal systems developed in our team, VoicePaint and NoteBook, are used to illustrate the discussion.
---
paper_title: Affective video content representation and modeling

paper_content:

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.
---
paper_title: Multimedia content processing through cross-modal association

paper_content:

Multimodal information processing has received considerable attention in recent years. The focus of existing research in this area has been predominantly on the use of fusion technology. In this paper, we suggest that cross-modal association can provide a new set of powerful solutions in this area. We investigate different cross-modal association methods using the linear correlation model. We also introduce a novel method for cross-modal association called Cross-modal Factor Analysis (CFA). Our earlier work on Latent Semantic Indexing (LSI) is extended for applications that use off-line supervised training. As a promising research direction and practical application of cross-modal association, cross-modal information retrieval where queries from one modality are used to search for content in another modality using low-level features is then discussed in detail. Different association methods are tested and compared using the proposed cross-modal retrieval system. All these methods achieve significant dimensionality reduction. Among them CFA gives the best retrieval performance. Finally, this paper addresses the use of cross-modal association to detect talking heads. The CFA method achieves 91.1% detection accuracy, while LSI and Canonical Correlation Analysis (CCA) achieve 66.1% and 73.9% accuracy, respectively. As shown by experiments, cross-modal association provides many useful benefits, such as robust noise resistance and effective feature selection. Compared to CCA and LSI, the proposed CFA shows several advantages in analysis performance and feature usage. Its capability in feature selection and noise resistance also makes CFA a promising tool for many multimedia analysis applications.
---
paper_title: Context and memory in multimedia content analysis

paper_content:

With the advent of broadband networking, video will be available online as well as through traditional distribution channels. The merging of entertainment and information media makes video content classification and retrieval a necessary tool. To provide fast retrieval, content management systems must discern between categories of video. Automatic multimedia analysis techniques for deriving high-level descriptions and annotations have experienced a tremendous surge in interest. Academia and industry have also been challenged to develop realistic applications-from home media library organizers and multimedia lecture archives to broadcast TV content navigators and video-on-demand-in pursuit of the killer application. Current content classification technologies have undoubtedly emerged from traditional image processing and computer vision, audio analysis and processing, and information retrieval. Although terminology varies, the algorithms generally fall into three categories: tangible detectors, high-level abstractors, and latent or intangible descriptors. This paper presents the reflections of the work done by the author and the work ahead.
---
paper_title: Multimodal Video Indexing: A Review of the State-of-the-art

paper_content:

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.
---
paper_title: A survey on content-based retrieval for multimedia databases

paper_content:

Conventional database systems are designed for managing textual and numerical data, and retrieving such data is often based on simple comparisons of text/numerical values. However, this simple method of retrieval is no longer adequate for multimedia data, since the digitized representation of images, video, or data itself does not convey the reality of these media items. In addition, composite data consisting of heterogeneous types of data also associates with the semantic content acquired by a user's recognition. Therefore, content-based retrieval for multimedia data is realized taking such intrinsic features of multimedia data into account. Implementation of the content-based retrieval facility is not based on a single fundamental, but is closely related to an underlying data model, a priori knowledge of the area of interest, and the scheme for representing queries. This paper surveys recent studies on content-based retrieval for multimedia databases from the point of view of three fundamental issues. Throughout the discussion, we assume databases that manage only nontextual/numerical data, such as image or video, are also in the category of multimedia databases.
---
paper_title: Multimedia content analysis-using both audio and visual clues

paper_content:

Multimedia content analysis refers to the computerized understanding of the semantic meanings of a multimedia document, such as a video sequence with an accompanying audio track. With a multimedia document, its semantics are embedded in multiple forms that are usually complimentary of each other, Therefore, it is necessary to analyze all types of data: image frames, sound tracks, texts that can be extracted from image frames, and spoken words that can be deciphered from the audio track. This usually involves segmenting the document into semantically meaningful units, classifying each unit into a predefined scene type, and indexing and summarizing the document for efficient retrieval and browsing. We review advances in using audio and visual information jointly for accomplishing the above tasks. We describe audio and visual features that can effectively characterize scene content, present selected algorithms for segmentation and classification, and review some testbed systems for video archiving and retrieval. We also describe audio and visual descriptors and description schemes that are being considered by the MPEG-7 standard for multimedia content description.
---
paper_title: Semantic annotation of images and videos for multimedia analysis

paper_content:

Annotations of multimedia documents typically have been pursued in two different directions. Either previous approaches have focused on low level descriptors, such as dominant color, or they have focused on the content dimension and corresponding annotations, such as person or vehicle. In this paper, we present a software environment to bridge between the two directions. M-OntoMat-Annotizer allows for linking low level MPEG-7 visual descriptions to conventional Semantic Web ontologies and annotations. We use M-OntoMat-Annotizer in order to construct ontologies that include prototypical instances of high-level domain concepts together with a formal specification of corresponding visual descriptors. Thus, we formalize the interrelationship of high- and low-level multimedia concept descriptions allowing for new kinds of multimedia content analysis and reasoning.
---
paper_title: SPATIAL ANALYSIS IN KEY-FRAME EXTRACTION USING VIDEO SEGMENTATION

paper_content:

Though being a vital part of video indexing and retrieval systems, key-frame extraction algorithms have been based mainly on the analysis of various frame similarities and their later clustering. This work broadens the spectra of the analysis by focusing on the spatio-temporal region relations present in the scene to determine the most representative frame in the shot. It applies efficient video segmentation to a low-resolution sequence representative utilising a novel iterative unsupervised segmentation algorithm based on the anisotropic diffusion paradigm and k-means clustering. Key-frames are determined by applying a set of heuristic rules to the behaviour and features of extracted spatio-temporal regions. Experimental results are given.
---
paper_title: Semantic indexing of multimedia documents

paper_content:

We propose two approaches for semantic indexing of audio-visual documents, based on bottom-up and top-down strategies. We base the first approach on a finite-state machine using low-level motion indices extracted from an MPEG compressed bitstream. The second approach innovatively performs semantic indexing through Hidden Markov Models.
---
paper_title: Relevance feedback in image retrieval: A comprehensive review

paper_content:

We analyze the nature of the relevance feedback problem in a continuous representation space in the context of content-based image retrieval. Emphasis is put on exploring the uniqueness of the problem and comparing the assumptions, implementations, and merits of various solutions in the literature. An attempt is made to compile a list of critical issues to consider when designing a relevance feedback algorithm. With a comprehensive review as the main portion, this paper also offers some novel solutions and perspectives throughout the discussion.
---
paper_title: SEGMENTING QUADRUPED GAIT PATTERNS FROM WILDLIFE VIDEO

paper_content:

This paper describes a novel approach to detecting walking quadrupeds in unedited wildlife film footage. Variable lighting, moving backgrounds and camouflaged animals make traditional foreground extraction techniques such as optical flow and background subtraction unstable. We track a sparse set of points over a short film clip and use RANSAC to segment the foreground region. A novel technique, employing normalised convolution is then utilised to interpolate dense flow from the sparsely defined foreground. Dense flow is extracted for a number of clips demonstrating quadruped gait and other movements. Principal component analysis (PCA) is applied to this set of dense flows and eigenvectors not encapsulating periodic internal motion characteristics are disregarded. The projection coefficients for the remaining principal components are analysed as one dimensional time series. Projection coefficient variation reflects changes in the velocity and relative alignment of the components of the foreground object. These coefficients' relative phase differences are deduced using spectral analysis and degree of periodicity using dynamic time warping. These parameters are used to train a KNN classifier which segments the training data with 93% success rate. By generating projection coefficients for unseen footage, the system has successfully located examples of quadruped gait previously missed by human observers.
---
paper_title: ICBR - Multimedia Management System for Intelligent Content Based Retrieval

paper_content:

This paper presents a system designed for the management of multimedia databases that embarks upon the problem of efficient media processing and representation for automatic semantic classification and modelling. Its objectives are founded on the integration of a large-scale wildlife digital media archive with a manually annotated semantic metadata organised in a structured taxonomy and media classification system. Novel techniques will be applied to temporal analysis, intelligent key-frame extraction, animal gait analysis, semantic modelling and audio classification. The system demonstrator will be developed as a part of an ICBR project within the 3C Research programme of convergent technology research for digital media processing and communications.
---
paper_title: On supervision and statistical learning for semantic multimedia analysis

paper_content:

Media analysis for video indexing is witnessing an increasing influence of statistical techniques. Examples of these techniques include the use of generative models as well as discriminant techniques for video structuring, classification, summarization, indexing, and retrieval. There is increasing emphasis on reducing the amount of supervision and user interaction needed to construct and utilize the semantic models. This paper highlights the statistical learning techniques in semantic multimedia indexing and retrieval. In particular the gamut of techniques from supervised to unsupervised systems will be demonstrated.
---
paper_title: Matching Words and Pictures

paper_content:

We present a new approach for modeling multi-modal data sets, focusing on the specific case of segmented images with associated text. Learning the joint distribution of image regions and words has many applications. We consider in detail predicting words associated with whole images (auto-annotation) and corresponding to particular image regions (region naming). Auto-annotation might help organize and access large collections of images. Region naming is a model of object recognition as a process of translating image regions to words, much as one might translate from one language to another. Learning the relationships between image regions and semantic correlates (words) is an interesting example of multi-modal data mining, particularly because it is typically hard to apply data mining techniques to collections of images. We develop a number of models for the joint distribution of image regions and words, including several which explicitly learn the correspondence between regions and words. We study multi-modal and correspondence extensions to Hofmann's hierarchical clustering/aspect model, a translation model adapted from statistical machine translation (Brown et al.), and a multi-modal extension to mixture of latent Dirichlet allocation (MoM-LDA). All models are assessed using a large collection of annotated images of real scenes. We study in depth the difficult problem of measuring performance. For the annotation task, we look at prediction performance on held out data. We present three alternative measures, oriented toward different types of task. Measuring the performance of correspondence methods is harder, because one must determine whether a word has been placed on the right region of an image. We can use annotation performance as a proxy measure, but accurate measurement requires hand labeled data, and thus must occur on a smaller scale. We show results using both an annotation proxy, and manually labeled data.
---
paper_title: Highlights modeling and detection in sports videos

paper_content:

Automatic annotation of semantic events allows effective retrieval of video content. In this work, we present solutions for highlights detection in sports videos. This application is particularly interesting for broadcasters, since they extensively use manual annotation to select interesting highlights that are edited to create new programmes. The proposed approach exploits the typical structure of a wide class of sports videos, namely, those related to sports which are played in delimited venues with playfields of well known geometry, like soccer, basketball, swimming, track and field disciplines, and so on. For this class of sports, a modeling scheme based on a limited set of visual cues and on finite state machines (FSM) that encode the temporal evolution of highlights is presented. Algorithms for model checking and for visual cues estimation are discussed, as well as applications of the representation to different sport domains.
---
paper_title: Automated Visual Recognition of Individual African Penguins

paper_content:

African penguins (Spheniscus demersus) carry a pattern of black spots on their chests that does not change from season to season during their adult life. Further, as far as we can tell, no two penguins have exactly the same pattern. We have developed a real-time system that can confidently locate African penguins whose chests are visible within video sequences or still images. An extraction of the chest spot pattern allows the generation of a unique biometrical identifier for each penguin. Using these identifiers an authentication of filmed or photographed African penguins against a population database can be performed. This paper provides a detailed technical description of the developed system and outlines the scope and the conditions of application
---
paper_title: TRACKING ANIMALS IN WILDLIFE VIDEOS USING FACE DETECTION

paper_content:

This paper presents an algorithm for detection and tracking of animal faces in wildlife videos. As an example the algorithm is applied to lion faces. The detection algorithm is based on a human face detection method, utilising Haar-like features and AdaBoost classifiers. The face tracking is implemented using the Kanade-Lucas-Tomasi tracker and by applying a specific interest model to the detected face. By combining the two methods in a specific tracking model, a reliable and temporally coherent detection/tracking of animal faces is achieved. In addition to the detection of particular animal species, the information generated by the tracker can be used to boost the priors in the probabilistic semantic classification of wildlife videos.
---
paper_title: Multimodal Video Indexing: A Review of the State-of-the-art

paper_content:

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.
---

Title: A Survey on Multimodal Video Representation for Semantic Retrieval

Section 1: INTRODUCTION
Description 1: Provide an introduction to the field of multimedia content-based retrieval, outlining the problems related to semantic gaps, the existing research paradigms, and emphasizing the need for advanced multimodal video representation.

Section 2: VIDEO REPRESENTATION
Description 2: Discuss the problem of computational video representation in the context of semantic retrieval, including foundational work and evolving paradigms such as Computational Media Aesthetics.

Section 3: MULTIMODALITY
Description 3: Elaborate on the concept of multimodality in digital video media, exploring how various communication channels and semiotic approaches contribute to semantic analysis and retrieval.

Section 4: DATA DRIVEN REPRESENTATIONS
Description 4: Examine the standard data-driven approaches for representing video media, including methodologies for low-level feature extraction and the resultant challenges in adapting to different domains.

Section 5: TOP-DOWN APPROACHES
Description 5: Explore top-down retrieval systems that utilize high-level domain knowledge to generate appropriate representations and discuss various techniques and methodologies within this approach.

Section 6: CONCLUSION
Description 6: Summarize the need for innovative video representation techniques to address the semantic gap, and highlight future directions towards the development of generic frameworks for indexing and retrieval of video content.