Video summarisation: A conceptual framework and survey of the state of the art
---
paper_title: An innovative algorithm for key frame extraction in video summarization

paper_content:

Video summarization, aimed at reducing the amount of data that must be examined in order to retrieve the information desired from information in a video, is an essential task in video analysis and indexing applications. We propose an innovative approach for the selection of representative (key) frames of a video sequence for video summarization. By analyzing the differences between two consecutive frames of a video sequence, the algorithm determines the complexity of the sequence in terms of changes in the visual content expressed by different frame descriptors. The algorithm, which escapes the complexity of existing methods based, for example, on clustering or optimization strategies, dynamically and rapidly selects a variable number of key frames within each sequence. The key frames are extracted by detecting curvature points within the curve of the cumulative frame differences. Another advantage is that it can extract the key frames on the fly: curvature points can be determined while computing the frame differences and the key frames can be extracted as soon as a second high curvature point has been detected. We compare the performance of this algorithm with that of other key frame extraction algorithms based on different approaches. The summaries obtained have been objectively evaluated by three quality measures: the Fidelity measure, the Shot Reconstruction Degree measure and the Compression Ratio measure.
---
paper_title: Content-based multimedia information retrieval: State of the art and challenges

paper_content:

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100p recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.
---
paper_title: Exploring video content structure for hierarchical summarization

paper_content:

In this paper, we propose a hierarchical video summarization strategy that explores video content structure to provide the users with a scalable, multilevel video summary. First, video-shot- segmentation and keyframe-extraction algorithms are applied to parse video sequences into physical shots and discrete keyframes. Next, an affinity (self-correlation) matrix is constructed to merge visually similar shots into clusters (supergroups). Since video shots with high similarities do not necessarily imply that they belong to the same story unit, temporal information is adopted by merging temporally adjacent shots (within a specified distance) from the supergroup into each video group. A video-scene-detection algorithm is thus proposed to merge temporally or spatially correlated video groups into scenario units. This is followed by a scene-clustering algorithm that eliminates visual redundancy among the units. A hierarchical video content structure with increasing granularity is constructed from the clustered scenes, video scenes, and video groups to keyframes. Finally, we introduce a hierarchical video summarization scheme by executing various approaches at different levels of the video content hierarchy to statically or dynamically construct the video summary. Extensive experiments based on real-world videos have been performed to validate the effectiveness of the proposed approach.
---
paper_title: Video caption detection and extraction using temporal information

paper_content:

Video caption detection and extraction is an important step for information retrieval in video databases. In this paper, we extract text information in video by fully utilizing the temporal information contained in the video. First we create a binary abstract sequence from a video segment. By analyzing the statistical pixel changes in the sequence, we can effectively locate the (dis)appealing frames of captions. Finally we extract the captions to create a summary of the video segment.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Context and memory in multimedia content analysis

paper_content:

With the advent of broadband networking, video will be available online as well as through traditional distribution channels. The merging of entertainment and information media makes video content classification and retrieval a necessary tool. To provide fast retrieval, content management systems must discern between categories of video. Automatic multimedia analysis techniques for deriving high-level descriptions and annotations have experienced a tremendous surge in interest. Academia and industry have also been challenged to develop realistic applications-from home media library organizers and multimedia lecture archives to broadcast TV content navigators and video-on-demand-in pursuit of the killer application. Current content classification technologies have undoubtedly emerged from traditional image processing and computer vision, audio analysis and processing, and information retrieval. Although terminology varies, the algorithms generally fall into three categories: tangible detectors, high-level abstractors, and latent or intangible descriptors. This paper presents the reflections of the work done by the author and the work ahead.
---
paper_title: Video summarization: methods and landscape

paper_content:

The ability to summarize and abstract information will be an essential part of intelligent behavior in consumer devices. Various summarization methods have been the topic of intensive research in the content-based video analysis community. Summarization in traditional information retrieval is a well understood problem. While there has been a lot of research in the multimedia community there is no agreed upon terminology and classification of the problems in this domain. Although the problem has been researched from different aspects there is usually no distinction between the various dimensions of summarization. The goal of the paper is to provide the basic definitions of widely used terms such as skimming, summarization, and highlighting. The different levels of summarization: local, global, and meta-level are made explicit. We distinguish among the dimensions of task, content, and method and provide an extensive classification model for the same. We map the existing summary extraction approaches in the literature into this model and we classify the aspects of proposed systems in the literature. In addition, we outline the evaluation methods and provide a brief survey. Finally we propose future research directions based on the white spots that we identified by analysis of existing systems in the literature.
---
paper_title: Doing Qualitative Research

paper_content:

PART ONE: INTRODUCTION How to Use This Book What You Can (and Can't) Do with Qualitative Research Introduction Why Do Students Use Qualitative Methods? Are Qualitative Methods Always the Best? Should You Use Qualitative Methods? Concluding Remarks The Research Experience I Introduction Moira's Research Diary Sally's Research Diary Simon's Research Diary Concluding Remarks The Research Experience II Introduction Interviews Ethnographies Texts Audio Data Visual Data Multiple Methods Concluding Remarks What Counts as 'Originality'? Introduction Originality Being a 'Professional' Independent Critical Thought Concluding Remarks PART TWO: STARTING OUT Selecting a Topic Introduction Workable Research Questions Simplistic Inductivism The 'Kitchen Sink' Gambit The Grand Theorist Strategies for Simplistic Inductivists Strategies for 'Kitchen Sinkers' Strategies for Grand Theorists Strategies for All Researchers Concluding Remarks Using Theories Introduction How Theoretical Models Shape Research The Different Languages of Qualitative Research Theories, Models and Hypotheses Examples Concluding Remarks Choosing a Methodology Introduction Qualitative or Quantitative? Your Research Strategy Choosing a Methodology: A Case Study Naturally Occurring Data? Multiple Methods? Concluding Remarks Selecting a Case Introduction What Is a Case Study? Generalizing from Cases Types of Case Studies Combining Qualitative Research with Quantitative Measures of Populations Purposive Sampling Theoretical Sampling Generalizability Is Present in a Single Case Concluding Remarks Ethical Research Introduction The Standards of Ethical Research Why Ethics Matter for Your Research Ethical Guidelines in Practice Complex Ethical Issues Research Governance Conclusion: Managing Ethical Demands Writing a Research Proposal Introduction Aim for Crystal Clarity Plan before You Write Be Persuasive Be Practical Make Broader Links Concluding Remarks PART THREE: COLLECTING AND ANALYZING YOUR DATA Collecting Your Data Collecting Interview Data Collecting Ethnographic Data Concluding Remarks Developing Data Analysis Introduction Kick-Starting Data Analysis A Case Study Interviews Fieldnotes Transcripts Visual Data Concluding Remarks Using Computers to Analyze Qualitative Data - Clive Seale Introduction What CAQDAS Software Can Do For You Advantages of CAQDAS Limitations and Disadvantages Theory Building With CAQDAS Keyword Analysis Concluding Remarks Quality in Qualitative Research Introduction Validity Reliability Concluding Remarks Evaluating Qualitative Research Introduction Two Guides For Evaluating Research Four Quality Criteria Applying Quality Criteria Four Quality Issues Revisited Concluding Remarks PART FOUR: WRITING UP The First Few Pages Introduction The Title The Abstract The List of Contents The Introduction Concluding Remarks The Literature Review Chapter Recording Your Reading Writing Your Literature Review Practical Questions Principles Do You Need A Literature Review Chapter? Concluding Remarks The Methodology Chapter Introduction What Should The Methodology Chapter Contain? A Natural History Chapter? Concluding Remarks Writing Your Data Chapters Introduction The Macrostructure The Microstructure Tightening Up Concluding Remarks Your Final Chapter Introduction The Final Chapter as Mutual Stimulation What Exactly Should Your Final Chapter Contain? Confessions And Trumpets Theorizing As Thinking Through Data Writing For Audiences Why Your Final Chapter Can Be Fun Concluding Remarks PART FIVE: GETTING SUPPORT Making Good Use of Your Supervisor Introduction Supervision Horror Stories Student And Supervisor Expectations The Early Stages The Later Stages Standards Of Good Practice Concluding Remarks Getting Feedback Introduction Writing Speaking The Art Of Presenting Research Feedback From The People You Study Concluding Remarks PART SIX: REVIEW Effective Qualitative Research Introduction Keep It Simple Take Advantage Of Using Qualitative Data Avoid Drowning In Data Avoid Journalism Concluding Remarks PART SEVEN: THE AFTERMATH Surviving an Oral Examination Introduction Viva Horror Stories Preparing For Your Oral Doing The Oral Outcomes Revising Your Thesis After The Oral A Case Study Concluding Remarks Getting Published Introduction The Backstage Politics Of Publishing Strategic Choices What Journals Are Looking For Reviewers' Comments How To Write A Short Journal Article Concluding Remarks Audiences Introduction The Policy-Making Audience The Practitioner Audience The Lay Audience Concluding Remarks Finding a Job Introduction Learning About Vacancies Getting On A Shortlist The Job Interview Concluding Remarks
---
paper_title: Qualitative Data Analysis: A User Friendly Guide for Social Scientists

paper_content:

From the Publisher: ::: Qualitative Data Analysis shows that learning how to analyse qualitative data by computer can be fun. Written in a stimulating style, with examples drawn mainly from every day life and contemporary humour, it should appeal to a wide audience.
---
paper_title: Context and memory in multimedia content analysis

paper_content:

With the advent of broadband networking, video will be available online as well as through traditional distribution channels. The merging of entertainment and information media makes video content classification and retrieval a necessary tool. To provide fast retrieval, content management systems must discern between categories of video. Automatic multimedia analysis techniques for deriving high-level descriptions and annotations have experienced a tremendous surge in interest. Academia and industry have also been challenged to develop realistic applications-from home media library organizers and multimedia lecture archives to broadcast TV content navigators and video-on-demand-in pursuit of the killer application. Current content classification technologies have undoubtedly emerged from traditional image processing and computer vision, audio analysis and processing, and information retrieval. Although terminology varies, the algorithms generally fall into three categories: tangible detectors, high-level abstractors, and latent or intangible descriptors. This paper presents the reflections of the work done by the author and the work ahead.
---
paper_title: Video summarization: methods and landscape

paper_content:

The ability to summarize and abstract information will be an essential part of intelligent behavior in consumer devices. Various summarization methods have been the topic of intensive research in the content-based video analysis community. Summarization in traditional information retrieval is a well understood problem. While there has been a lot of research in the multimedia community there is no agreed upon terminology and classification of the problems in this domain. Although the problem has been researched from different aspects there is usually no distinction between the various dimensions of summarization. The goal of the paper is to provide the basic definitions of widely used terms such as skimming, summarization, and highlighting. The different levels of summarization: local, global, and meta-level are made explicit. We distinguish among the dimensions of task, content, and method and provide an extensive classification model for the same. We map the existing summary extraction approaches in the literature into this model and we classify the aspects of proposed systems in the literature. In addition, we outline the evaluation methods and provide a brief survey. Finally we propose future research directions based on the white spots that we identified by analysis of existing systems in the literature.
---
paper_title: Optimizing user expectations for video semantic filtering and abstraction

paper_content:

We describe a novel automatic system that generates personalized videos based on semantic filtering or summarization techniques. This system uses a new set of more than one hundred visual semantic detectors that automatically detect video concepts in faster than realtime. Based on personal profiles, the system generates either video summaries from video databases or filtered video contents from live broadcasting videos. The prototype experiments have shown the effectiveness and stabilities of the system.
---
paper_title: Video summarization based on user log enhanced link analysis

paper_content:

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
---
paper_title: Tennis video abstraction from audio and visual cues

paper_content:

We propose a context-based model of video abstraction exploiting both audio and video features and applied to tennis TV programs. We can automatically produce different types of summary of a given video depending on the users' constraints or preferences. We have first designed an efficient and accurate temporal segmentation of the video into segments homogeneous w.r.t the camera motion. We introduce original visual descriptors related to the dominant and residual image motions. The different summary types are obtained by specifying adapted classification criteria which involve audio features to select the relevant segments to be included in the video abstract. The proposed scheme has been validated on 22 hours of tennis videos.
---
paper_title: From context to content: leveraging context to infer media metadata

paper_content:

The recent popularity of mobile camera phones allows for new opportunities to gather important metadata at the point of capture. This paper describes a method for generating metadata for photos using spatial, temporal, and social context. We describe a system we implemented for inferring location information for pictures taken with camera phones and its performance evaluation. We propose that leveraging contextual metadata at the point of capture can address the problems of the semantic and sensory gaps. In particular, combining and sharing spatial, temporal, and social contextual metadata from a given user and across users allows us to make inferences about media content.
---
paper_title: Automatic Soccer Video Analysis and Summarization

paper_content:

We propose a fully automatic and computationally efficient framework for analysis and summarization of soccer videos using cinematic and object-based features. The proposed framework includes some novel low-level processing algorithms, such as dominant color region detection, robust shot boundary detection, and shot classification, as well as some higher-level algorithms for goal detection, referee detection, and penalty-box detection. The system can output three types of summaries: i) all slow-motion segments in a game; ii) all goals in a game; iii) slow-motion segments classified according to object-based features. The first two types of summaries are based on cinematic features only for speedy processing, while the summaries of the last type contain higher-level semantics. The proposed framework is efficient, effective, and robust. It is efficient in the sense that there is no need to compute object-based features when cinematic features are sufficient for the detection of certain events, e.g., goals in soccer. It is effective in the sense that the framework can also employ object-based features when needed to increase accuracy (at the expense of more computation). The efficiency, effectiveness, and robustness of the proposed framework are demonstrated over a large data set, consisting of more than 13 hours of soccer video, captured in different countries and under different conditions.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Design and evaluation of a music video summarization system

paper_content:

We present a system that summarizes the textual, audio, and video information of music videos in a format tuned to the preferences of a focus group of 20 users. First, we analyzed user-needs for the content and the layout of the music summaries. Then, we designed algorithms that segment individual song videos from full music video programs by noting changes in color palette, transcript, and audio classification. We summarize each song with automatically selected high level information such as title, artist, duration, title frame, and text as well as audio and visual segments of the chorus. Our system automatically determines with high recall and precision chorus locations, from the placement of repeated words and phrases in the text of the song's lyrics. Our Bayesian belief network then selects other significant video and audio content from the multiple media. Overall, we are able to compress content by a factor of 10. Our second user study has identified the principal variations between users in their choices of content desired in the summary, and in their choices of the platforms that should support their viewing.
---
paper_title: Summarizing wearable video

paper_content:

"We want to record our entire life by video" is the motivation of this research. Developing wearable devices and huge storage devices will make it possible to keep entire life by video. We could capture 70 years of our life, however, the problem is how to handle such a huge amount of data. Automatic summarization based on personal interest should be required. In this paper we propose an approach to the automatic structuring and summarization of wearable video. (Wearable video is our abbreviation of "video captured by a wearable camera".) In our approach, we make use of a wearable camera and a sensor of brain waves. The video is firstly structured by objective features of video, and the shots are rated by subjective measures based on brain waves. The approach is very successful for real world experiments and it automatically extracted all the events that the subjects reported they had felt interesting.
---
paper_title: Creating audio keywords for event detection in soccer video

paper_content:

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.
---
paper_title: Efficient retrieval of life log based on context and content

paper_content:

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.
---
paper_title: Event detection in baseball video using superimposed caption recognition

paper_content:

We have developed a novel system for baseball video event detection and summarization using superimposed caption text detection and recognition. The system detects different types of semantic level events in baseball video including scoring and last pitch of each batter. The system has two components: event detection and event boundary detection. Event detection is realized by change detection and recognition of game stat texts (such as text information showing in score box). Event boundary detection is achieved using our previously developed algorithm, which detects the pitch view as the event beginning and nonactive view as potential endings of the event. One unique contribution of the system is its capability to accurately detect the semantic level events by combining video text recognition with camera view recognition. Another unique feature is the real-time processing speed by taking advantage of compressed-domain approaches in part of the algorithms such as caption detection. To the best of our knowledge, this is the first system achieving accurate detection of multiple types of high-level semantic events in baseball videos.
---
paper_title: The priority curve algorithm for video summarization

paper_content:

In this paper, we introduce the concept of a priority curve associated with a video. We then provide an algorithm that can use the priority curve to create a summary (of a desired length) of any video. The summary thus created exhibits nice continuity properties and also avoids repetition. We have implemented the priority curve algorithm (PCA) and compared it with other summarization algorithms in the literature. We show that PCA is faster than existing algorithms and also produces better quality summaries. The quality of summaries was evaluated by a group of 200 students in Naples, Italy, who watched soccer videos. We also briefly describe a soccer video summarization system we have built on using the PCA architecture and various (classical) image processing algorithms.
---
paper_title: Multimedia content analysis-using both audio and visual clues

paper_content:

Multimedia content analysis refers to the computerized understanding of the semantic meanings of a multimedia document, such as a video sequence with an accompanying audio track. With a multimedia document, its semantics are embedded in multiple forms that are usually complimentary of each other, Therefore, it is necessary to analyze all types of data: image frames, sound tracks, texts that can be extracted from image frames, and spoken words that can be deciphered from the audio track. This usually involves segmenting the document into semantically meaningful units, classifying each unit into a predefined scene type, and indexing and summarizing the document for efficient retrieval and browsing. We review advances in using audio and visual information jointly for accomplishing the above tasks. We describe audio and visual features that can effectively characterize scene content, present selected algorithms for segmentation and classification, and review some testbed systems for video archiving and retrieval. We also describe audio and visual descriptors and description schemes that are being considered by the MPEG-7 standard for multimedia content description.
---
paper_title: Generation of personalized abstract of sports video

paper_content:

Video abstraction is defined as creating a shorter video clip from an original video stream. In this paper, we propose a method of generating a personalized abstract of broadcasted sports video. We first detect significant events from the video stream by matching with gamestats in which highlights of the game are described. Textual information in an overlay appearing on an image frame is recognized for this matching. Then, we select highlight shots from these detected events, reflecting on personal preferences. Finally, we connect each shot augmented with related audio and text in temporal order. From experimental results, we verified that an hourlength video can be compressed into a minute-length personalized abstract.
---
paper_title: MSN: statistical understanding of broadcasted baseball video using multi-level semantic network

paper_content:

The information processing of sports video yields valuable semantics for content delivery over narrowband networks. Traditional image/video processing is formulated in terms of low-level features describing image/video structure and intensity, while the high-level knowledge such as common sense and human perceptual knowledge are encoded in abstract and nongeometric representations. The management of semantic information in video becomes more and more difficult because of the large difference in representations, levels of knowledge, and abstract episodes. This paper proposes a semantic highlight detection scheme using a Multi-level Semantic Network (MSN) for baseball video interpretation. The probabilistic structure can be applied for highlight detection and shot classification. Satisfactory results will be shown to illustrate better performance compared with the traditional ones.
---
paper_title: Highlights for more complete sports video summarization

paper_content:

Summarization is an essential requirement for achieving a more compact and interesting representation of sports video contents. We propose a framework that integrates highlights into play segments and reveal why we should still retain breaks. Experimental results show that fast detections of whistle sounds, crowd excitement, and text boxes can complement existing techniques for play-breaks and highlights localization.
---
paper_title: Interface Design for MyInfo: a Personal News Demonstrator Combining Web and TV Content

paper_content:

This paper describes MyInfo, a novel interface for a personal news demonstrator that processes and combines content from TV and the web. We detail our design process from concept generation to focus group exploration to final design. Our design focuses on three issues: (i) ease-of-use, (ii) video summarization, and (iii) news personalization. With a single button press on the remote control, users access specific topics such as weather or traffic. In addition, users can play back personalized news content as a TV show, leaving themselves free to complete other tasks in their homes, while consuming the news.
---
paper_title: Content-based multimedia information retrieval: State of the art and challenges

paper_content:

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100p recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.
---
paper_title: Video summarization based on user log enhanced link analysis

paper_content:

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
---
paper_title: Two-Stage Hierarchical Video Summary Extraction to Match Low-Level User Browsing Preferences

paper_content:

A compact summary of video that conveys visual content at various levels of detail enhances user interaction significantly. In this paper, we propose a two-stage framework to generate MPEG-7-compliant hierarchical key frame summaries of video sequences. At the first stage, which is carried out off-line at the time of content production, fuzzy clustering and data pruning methods are applied to given video segments to obtain a nonredundant set of key frames that comprise the finest level of the hierarchical summary. The number of key frames allocated to each shot or segment is determined dynamically and without user supervision through the use of cluster validation techniques. A coarser summary is generated on-demand in the second stage by reducing the number of key frames to match the low-level browsing preferences of a user. The proposed method has been validated by experimental results on a collection of video programs.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Efficient retrieval of life log based on context and content

paper_content:

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.
---
paper_title: Using MPEG-7 and MPEG-21 for personalizing video

paper_content:

As multimedia content has proliferated over the past several years, users have begun to expect that content be easily accessed according to their own preferences. One of the most effective ways to do this is through using the MPEG-7 and MPEG-21 standards, which can help address the issues associated with designing a video personalization and summarization system in heterogeneous usage environments. This three-tier architecture provides a standards-compliant infrastructure that, in conjunction with our tools, can help select, adapt, and deliver personalized video summaries to users. In extending our summarization research, we plan to explore semantic similarities across multiple simultaneous news media sources and to abstract summaries for different viewpoints. Doing so will allow us to track a semantic topic as it evolves into the future. As a result, we should be able to summarize news repositories into a smaller collection of topic threads.
---
paper_title: Automatic generation of video summaries for historical films

paper_content:

A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. In the ECHO (European Chronicles On-line) project, a system was developed to store and manage large collections of historical films for the preservation of cultural heritage. At the University of Mannheim, we have developed the video summarization component of the ECHO system. We discuss the particular challenges the historical film material poses, and how we have designed new video processing algorithms and modified existing ones to cope with noisy black-and-white films.
---
paper_title: Optimizing user expectations for video semantic filtering and abstraction

paper_content:

We describe a novel automatic system that generates personalized videos based on semantic filtering or summarization techniques. This system uses a new set of more than one hundred visual semantic detectors that automatically detect video concepts in faster than realtime. Based on personal profiles, the system generates either video summaries from video databases or filtered video contents from live broadcasting videos. The prototype experiments have shown the effectiveness and stabilities of the system.
---
paper_title: A highlight scene detection and video summarization system using audio feature for a personal video recorder

paper_content:

The personal video recorder such as recordable-DVD recorder, Blu-ray disc recorder and/or hard disc recorder has become popular for a large volume storage device for video/audio content data and a browsing function that would quickly provide a desired scene to the user is required as an essential part of such a large capacity recording/playback system. We propose a highlight scene detection function by using only 'audio' features and realize a browsing function for the recorder that enables completely automatic detection of sports highlights. We detect sports highlights by identifying portions with "commentator's excited speech" using Gaussian mixture models (GMM's) trained using the MDL criterion. Our computation is carried out directly on the MDCT coefficients from the AC-3 coefficients thus giving us a tremendous speed advantage. Our accuracy of detection of sports highlights is high across a variety of sports.
---
paper_title: An empirical investigation into user navigation of digital video using the VCR-like control set

paper_content:

There has been an almost explosive growth in digital video in recent years. The convention for enabling users to navigate digital video is the Video Cassette Recorder-like (VCR-like) control set, which is dictated by the proliferation of media players that embody it, including Windows Media Player and QuickTime. However, there is a dearth of research seeking to understand how users relate to this control set and how useful it actually is in practice. This paper details our empirical investigation of the issue. A digital video navigation system with a VCR-like control set was developed and subsequently used by a large sample of users (n=200), who were required to complete a number of goal-directed navigational tasks. Each user's navigational activity was tracked and recorded automatically by the system. Analysis of the navigational data revealed a range of results concerning how the VCR-like control set both enhanced and limited the user's ability to locate sequences of interest, including a number of searching and browsing strategies that were exploited by the users.
---
paper_title: Video Adaptation: Concepts, Technologies, and Open Issues

paper_content:

Video adaptation is an emerging field that offers a rich body of techniques for answering challenging questions in pervasive media applications. It transforms the input video(s) to an output in video or augmented multimedia form by utilizing manipulations at multiple levels (signal, structural, or semantic) in order to meet diverse resource constraints and user preferences while optimizing the overall utility of the video. There has been a vast amount of activity in research and standard development in this area. This paper first presents a general framework that defines the fundamental entities, important concepts (i.e., adaptation, resource, and utility), and formulation of video adaptation as constrained optimization problems. A taxonomy is used to classify different types of adaptation techniques. The state of the art in several active research areas is reviewed with open challenging issues identified. Finally, support of video adaptation from related international standards is discussed.
---
paper_title: Using MPEG-7 and MPEG-21 for personalizing video

paper_content:

As multimedia content has proliferated over the past several years, users have begun to expect that content be easily accessed according to their own preferences. One of the most effective ways to do this is through using the MPEG-7 and MPEG-21 standards, which can help address the issues associated with designing a video personalization and summarization system in heterogeneous usage environments. This three-tier architecture provides a standards-compliant infrastructure that, in conjunction with our tools, can help select, adapt, and deliver personalized video summaries to users. In extending our summarization research, we plan to explore semantic similarities across multiple simultaneous news media sources and to abstract summaries for different viewpoints. Doing so will allow us to track a semantic topic as it evolves into the future. As a result, we should be able to summarize news repositories into a smaller collection of topic threads.
---
paper_title: Interface Design for MyInfo: a Personal News Demonstrator Combining Web and TV Content

paper_content:

This paper describes MyInfo, a novel interface for a personal news demonstrator that processes and combines content from TV and the web. We detail our design process from concept generation to focus group exploration to final design. Our design focuses on three issues: (i) ease-of-use, (ii) video summarization, and (iii) news personalization. With a single button press on the remote control, users access specific topics such as weather or traffic. In addition, users can play back personalized news content as a TV show, leaving themselves free to complete other tasks in their homes, while consuming the news.
---
paper_title: Video skimming and characterization through the combination of image and language understanding

paper_content:

Digital video is rapidly becoming important for education, entertainment and a host of multimedia applications. With the size of the video collections growing to thousands of hours, technology is needed to effectively browse segments in a short time without losing the content of the video. We propose a method to extract the significant audio and video information and create a skim video which represents a very short synopsis of the original. The goal of this work is to show the utility of integrating language and image understanding techniques for video skimming by extraction of significant information, such as specific objects, audio keywords and relevant video structure. The resulting skim video is much shorter; where compaction is as high as 20:1, and yet retains the essential content of the original segment. We have conducted a user-study to test the content summarization and effectiveness of the skim as a browsing tool.
---
paper_title: An innovative algorithm for key frame extraction in video summarization

paper_content:

Video summarization, aimed at reducing the amount of data that must be examined in order to retrieve the information desired from information in a video, is an essential task in video analysis and indexing applications. We propose an innovative approach for the selection of representative (key) frames of a video sequence for video summarization. By analyzing the differences between two consecutive frames of a video sequence, the algorithm determines the complexity of the sequence in terms of changes in the visual content expressed by different frame descriptors. The algorithm, which escapes the complexity of existing methods based, for example, on clustering or optimization strategies, dynamically and rapidly selects a variable number of key frames within each sequence. The key frames are extracted by detecting curvature points within the curve of the cumulative frame differences. Another advantage is that it can extract the key frames on the fly: curvature points can be determined while computing the frame differences and the key frames can be extracted as soon as a second high curvature point has been detected. We compare the performance of this algorithm with that of other key frame extraction algorithms based on different approaches. The summaries obtained have been objectively evaluated by three quality measures: the Fidelity measure, the Shot Reconstruction Degree measure and the Compression Ratio measure.
---
paper_title: Highlight sound effects detection in audio stream

paper_content:

This paper addresses the problem of highlight sound effects detection in audio stream, which is very useful in fields of video summarization and highlight extraction. Unlike researches on audio segmentation and classification, in this domain, it just locates those highlight sound effects in audio stream. An extensible framework is proposed and in current system three sound effects are considered: laughter, applause and cheer, which are tied up with highlight events in entertainments, sports, meetings and home videos. HMMs are used to model these sound effects and a log-likelihood scores based method is used to make final decision. A sound effect attention model is also proposed to extend general audio attention model for highlight extraction and video summarization. Evaluations on a 2-hours audio database showed very encouraging results.
---
paper_title: Classification of self-consumable highlights for soccer video summaries

paper_content:

An effective scheme for soccer summarization is significant to improve the usage of this massively growing video data. The paper presents an extension to our recent work which proposed a framework to integrate highlights into play-breaks to construct more complete soccer summaries. The current focus is to demonstrate the benefits of detecting some specific audio-visual features during play-break sequences in order to classify highlights contained within them. The main purpose is to generate summaries which are self-consumable individually. To support this framework, the algorithms for shot classification and detection of near-goal and slow-motion replay scenes is described. The results of our experiment using 5 soccer videos (20 minutes each) show the performance and reliability of our framework
---
paper_title: Augmented segmentation and visualization for presentation videos

paper_content:

We investigate methods of segmenting, visualizing, and indexing presentation videos by both audio and visual data. The audio track is segmented by speaker, and augmented with key phrases which are extracted using an Automatic Speech Recognizer (ASR). The video track is segmented by visual dissimilarities and changes in speaker gesturing, and augmented by representative key frames. An interactive user interface combines a visual representation of audio, video, text, key frames, and allows the user to navigate presentation videos. User studies with 176 students of varying knowledge were conducted on 7.5 hours of student presentation video (32 presentations). Tasks included searching for various portions of presentations, both known and unknown to students, and summarizing presentations given the annotations. The results are favorable towards the video summaries and the interface, suggesting faster responses by a factor of 20% compared to having access to the actual video. Accuracy of responses remained the same on average. Follow-up surveys present a number of suggestions towards improving the interface, such as the incorporation of automatic speaker clustering and identification, and the display of an abstract topological view of the presentation. Surveys also show alternative contexts in which students would like to use the tool in the classroom environment.
---
paper_title: A fast layout algorithm for visual video summaries

paper_content:

We created an improved layout algorithm for automatically generating visual video summaries reminiscent of comic book pages. The summaries are comprised of images from the video that are sized according to their importance. The algorithm performs a global optimization with respect to a layout cost function that encompasses features such as the number of resized images and the amount of whitespace in the presentation. The algorithm creates summaries that: always fit exactly into the requested area, are varied by containing few rows with images of the same size, and have little whitespace at the end of the last row. The layout algorithm is fast enough to allow the interactive resizing of the summaries and the subsequent generation of a new layout.
---
paper_title: Spatio-temporal quality assessment for home videos

paper_content:

Compared with the video programs taken by professionals, home videos are always with low-quality content resulted from lack of professional capture skills. In this paper, we present a novel spatio-temporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, sub-shot, is selected as the basic unit for quality assessment. A set of spatio-temporal artifacts, regarded as the key factors affecting the overall perceived quality (i.e. unstableness, jerkiness, infidelity, blurring, brightness and orientation), are mined from each sub-shot based on the particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three different methods, including user study, factor fusion, and a learning-based scheme. To validate the proposed scheme, we present a scalable quality-based home video summarization system, aiming at achieving the best quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model based video skimming approach demonstrated the effectiveness of the proposed quality assessment scheme.
---
paper_title: Video summarization by spatial-temporal graph optimization

paper_content:

In this paper we present a novel approach for video summarization based on graph optimization. Our approach emphasizes both a comprehensive visual-temporal content coverage and visual coherence of the video summary. The approach has three stages. First, the source video is segmented into video shots, and a candidate shot set is selected from the video shots according to some video features. Second, a dissimilarity function is defined between the video shots to describe their spatial-temporal relation, and the candidate video shot set is modelled into a directional graph. Third, we outline a dynamic programming algorithm and use it to search the longest path in the graph as the final video skimming. A static video summary is generated at the same time. Experimental results show encouraging promises of our approach for video summarization.
---
paper_title: Affective video content representation and modeling

paper_content:

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.
---
paper_title: Video caption detection and extraction using temporal information

paper_content:

Video caption detection and extraction is an important step for information retrieval in video databases. In this paper, we extract text information in video by fully utilizing the temporal information contained in the video. First we create a binary abstract sequence from a video segment. By analyzing the statistical pixel changes in the sequence, we can effectively locate the (dis)appealing frames of captions. Finally we extract the captions to create a summary of the video segment.
---
paper_title: Automatic Soccer Video Analysis and Summarization

paper_content:

We propose a fully automatic and computationally efficient framework for analysis and summarization of soccer videos using cinematic and object-based features. The proposed framework includes some novel low-level processing algorithms, such as dominant color region detection, robust shot boundary detection, and shot classification, as well as some higher-level algorithms for goal detection, referee detection, and penalty-box detection. The system can output three types of summaries: i) all slow-motion segments in a game; ii) all goals in a game; iii) slow-motion segments classified according to object-based features. The first two types of summaries are based on cinematic features only for speedy processing, while the summaries of the last type contain higher-level semantics. The proposed framework is efficient, effective, and robust. It is efficient in the sense that there is no need to compute object-based features when cinematic features are sufficient for the detection of certain events, e.g., goals in soccer. It is effective in the sense that the framework can also employ object-based features when needed to increase accuracy (at the expense of more computation). The efficiency, effectiveness, and robustness of the proposed framework are demonstrated over a large data set, consisting of more than 13 hours of soccer video, captured in different countries and under different conditions.
---
paper_title: Dynamic storyboards for video content summarization

paper_content:

We propose an innovative, general purpose, approach to the selection and hierarchical representation of key frames of a video sequence for video summarization. In the first stage the shot detection module performs the video structural analysis; in the second stage the key frame extraction module creates the visual summary; and the last stage the summary post-processing module, after a pre-a classification aimed remove meaningless key frames, create a multilevel storyboard that the user may browse.
---
paper_title: Two-Stage Hierarchical Video Summary Extraction to Match Low-Level User Browsing Preferences

paper_content:

A compact summary of video that conveys visual content at various levels of detail enhances user interaction significantly. In this paper, we propose a two-stage framework to generate MPEG-7-compliant hierarchical key frame summaries of video sequences. At the first stage, which is carried out off-line at the time of content production, fuzzy clustering and data pruning methods are applied to given video segments to obtain a nonredundant set of key frames that comprise the finest level of the hierarchical summary. The number of key frames allocated to each shot or segment is determined dynamically and without user supervision through the use of cluster validation techniques. A coarser summary is generated on-demand in the second stage by reducing the number of key frames to match the low-level browsing preferences of a user. The proposed method has been validated by experimental results on a collection of video programs.
---
paper_title: A highlight scene detection and video summarization system using audio feature for a personal video recorder

paper_content:

The personal video recorder such as recordable-DVD recorder, Blu-ray disc recorder and/or hard disc recorder has become popular for a large volume storage device for video/audio content data and a browsing function that would quickly provide a desired scene to the user is required as an essential part of such a large capacity recording/playback system. We propose a highlight scene detection function by using only 'audio' features and realize a browsing function for the recorder that enables completely automatic detection of sports highlights. We detect sports highlights by identifying portions with "commentator's excited speech" using Gaussian mixture models (GMM's) trained using the MDL criterion. Our computation is carried out directly on the MDCT coefficients from the AC-3 coefficients thus giving us a tremendous speed advantage. Our accuracy of detection of sports highlights is high across a variety of sports.
---
paper_title: Flashlight and player detection in fighting sport for video summarization

paper_content:

In this paper, we present a method for generating fighting sport videos summary highlights using flashlight [N. Benjamas et al., May 2005] and player detection by detecting frames that contain close-up flashlight and players. Detected flashlight and distance between players is utilized in efficient summarization of fighting sport videos. The proposed algorithm first player was detected using skin color detection and connected component labeling. Then, the algorithm identifies the correct frame by calculating the distance between players that distance between one player and another one. Our algorithm accurately detects distance between players that it has the ability to capture inherently important events.
---
paper_title: A utility framework for the automatic generation of audio-visual skims

paper_content:

In this paper, we present a novel algorithm for generating audio-visual skims from computable scenes. Skims are useful for browsing digital libraries, and for on-demand summaries in set-top boxes. A computable scene is a chunk of data that exhibits consistencies with respect to chromaticity, lighting and sound. There are three key aspects to our approach: (a) visual complexity and grammar, (b) robust audio segmentation and (c) an utility model for skim generation. We define a measure of visual complexity of a shot, and map complexity to the minimum time for comprehending the shot. Then, we analyze the underlying visual grammar, since it makes the shot sequence meaningful. We segment the audio data into four classes, and then detect significant phrases in the speech segments. The utility functions are defined in terms of complexity and duration of the segment. The target skim is created using a general constrained utility maximization procedure that maximizes the information content and the coherence of the resulting skim. The objective function is constrained due to multimedia synchronization constraints, visual syntax and by penalty functions on audio and video segments. The user study results indicate that the optimal skims show statistically significant differences with other skims with compression rates up to 90%.
---
paper_title: Adaptive extraction of highlights from a sport video based on excitement modeling

paper_content:

This paper addresses the challenge of automatically extracting the highlights from sports TV broadcasts. In particular, we are interested in finding a generic method of highlights extraction, which does not require the development of models for the events that are thought to be interpreted by the users as highlights. Instead, we search for highlights in those video segments that are expected to excite the users most. It is namely realistic to assume that a highlighting event induces a steady increase in a user's excitement, as compared to other, less interesting events. We mimic the expected variations in a user's excitement by observing the temporal behavior of selected audiovisual low-level features and the editing scheme of a video. Relations between this noncontent information and the evoked excitement are drawn partly from psychophysiological research and partly from analyzing the live-video directing practice. The expected variations in a user's excitement are represented by the excitement time curve, which is, subsequently, filtered in an adaptive way to extract the highlights in the prespecified total length and in view of the preferences regarding the highlights strength: extraction can namely be performed with variable sensitivity to capture few "strong" highlights or more "less strong" ones. We evaluate and discuss the performance of our method on the case study of soccer TV broadcasts.
---
paper_title: Semantic units detection and summarization of baseball videos

paper_content:

A framework for analyzing baseball videos and generation of game summary is proposed. Due to the well-defined rules of baseball games, the system efficiently detects semantic units by the domain-related knowledge, and therefore, automatically discovers the structure of a baseball game. After extracting the information changes that are caused by some semantic events on the superimposed caption, a rule-based decision tree is applied to detect meaningful events. Only three types of information, including number-of-outs, score, and base occupation status, are taken in the detection process, and thus the framework detects events and produces summarization in an efficient and effective manner. The experimental results show the effectiveness of this framework and some research opportunities about generating semantic-level summary for sports videos.
---
paper_title: Information theory-based shot cut/fade detection and video summarization

paper_content:

New methods for detecting shot boundaries in video sequences and for extracting key frames using metrics based on information theory are proposed. The method for shot boundary detection relies on the mutual information (MI) and the joint entropy (JE) between the frames. It can detect cuts, fade-ins and fade-outs. The detection technique was tested on the TRECVID2003 video test set having different types of shots and containing significant object and camera motion inside the shots. It is demonstrated that the method detects both fades and abrupt cuts with high accuracy. The information theory measure provides us with better results because it exploits the inter-frame information in a more compact way than frame subtraction. It was also successfully compared to other methods published in literature. The method for key frame extraction uses MI as well. We show that it captures satisfactorily the visual content of the shot.
---
paper_title: Creating audio keywords for event detection in soccer video

paper_content:

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.
---
paper_title: Automatically generating summaries for musical video

paper_content:

In this paper, we propose a novel approach to automatically summarize musical videos. The proposed summarization scheme is different from the current methods used for video summarization. The musical video is separated into the musical and visual tracks. A music summary is created by analyzing the music content based on music features, adaptive clustering algorithm and musical domain knowledge. Then, shots are detected and clustered in the visual track. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of summarization. The experiments on different genres of musical video and comparisons with the summaries only based on music track and video track indicate that the results of summarization using proposed method are significant and effective to help realize user's expectation.
---
paper_title: Event detection in baseball video using superimposed caption recognition

paper_content:

We have developed a novel system for baseball video event detection and summarization using superimposed caption text detection and recognition. The system detects different types of semantic level events in baseball video including scoring and last pitch of each batter. The system has two components: event detection and event boundary detection. Event detection is realized by change detection and recognition of game stat texts (such as text information showing in score box). Event boundary detection is achieved using our previously developed algorithm, which detects the pitch view as the event beginning and nonactive view as potential endings of the event. One unique contribution of the system is its capability to accurately detect the semantic level events by combining video text recognition with camera view recognition. Another unique feature is the real-time processing speed by taking advantage of compressed-domain approaches in part of the algorithms such as caption detection. To the best of our knowledge, this is the first system achieving accurate detection of multiple types of high-level semantic events in baseball videos.
---
paper_title: An approach to generating two-level video abstraction

paper_content:

Video abstraction is a short summary of the content of a longer video document. Most existing video abstraction methods are based on shot-level, which is not sufficient to meaningful browsing and is too fine to users sometimes. In this paper, we propose a novel approach of generating video abstraction at two levels, namely, the shot-level and the scene-level. We put up a method of extracting key frames from shots, according to the content variation of the latter. An updated time-adaptive algorithm is used to group the shots into scene and representative frames are extracted in the region of that scene using the method of generating Minimum Spanning Tree. Key frames and representative frames can represent the content of shots and scenes, respectively. The organized sequences of key frames and representative frames are the two-level video abstraction. Experiments based on real-world movies show that the method above can provide users with better video summary at different levels.
---
paper_title: Summarization of news video and its description for content‐based access

paper_content:

A video summary abstracts the entirety with the gist without losing the essential content of the original video and also facilitates efficient content-based access to the desired content. In this article, we propose a novel method for summarizing a news video based on multimodal analysis of the content. The proposed method exploits the closed caption (CC) data to locate semantically meaningful highlights in a news video and speech signals in an audio stream to align the CC data with the video in a time-line. Then, the extracted highlights are described in a multilevel structure using the MPEG-7 Summarization Description Scheme (DS). Specifically, we use the HierarchicalSummary DS that allows efficient accessing of the content through such functionalities as multilevel abstracts and navigation guidance in a hierarchical fashion. Intensive experiments with our prototypical systems are presented to demonstrate the validity and reliability of the proposed method in real applications. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13, 267–274, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10067
---
paper_title: Narrative abstraction model for story-oriented video

paper_content:

TV program review services, especially drama review services, are one of the most popular video on demand services on the Web. In this paper, we propose a novel video abstraction model for a review service of story-oriented video such as dramas. In a drama review service, viewers want to understand the story in a short time and service providers want to provide video abstracts at minimum cost. The proposed model enables the automatic creation of a video abstract that still allows viewers to understand the overall story of the source video. Also, the model has a flexible structure so that the duration of an abstract can be adjusted depending on the requirements given by viewers. We get clues for human understanding of a story from scenario writing rules and editorial techniques which are popularly used in the process of video producing. We have implemented the proposed model and successfully applied it to several TV dramas.
---
paper_title: Automatically extracting highlights for TV Baseball programs

paper_content:

In today's fast-paced world, while the number of channels of television programming available is increasing rapidly, the time available to watch them remains the same or is decreasing. Users desire the capability to watch the programs time-shifted (on-demand) and/or to watch just the highlights to save time. In this paper we explore how to provide for the latter capability, that is the ability to extract highlights automatically, so that viewing time can be reduced. We focus on the sport of baseball as our initial target—it is a very popular sport, the whole game is quite long, and the exciting portions are few. We focus on detecting highlights using audio-track features alone without relying on expensive-to-compute video-track features. We use a combination of generic sports features and baseball-specific features to obtain our results, but believe that may other sports offer the same opportunity and that the techniques presented here will apply to those sports. We present details on relative performance of various learning algorithms, and a probabilistic framework for combining multiple sources of information. We present results comparing output of our algorithms against human-selected highlights for a diverse collection of baseball games with very encouraging results.
---
paper_title: Sequential association mining for video summarization

paper_content:

In this paper, we propose an association-based video summarization scheme that mines sequential associations from video data for summary creation. Given detected shots of video V, we first cluster them into visually distinct groups, and then construct a sequential sequence by integrating the temporal order and cluster type of each shot. An association mining scheme is designed to mine sequentially associated clusters from the sequence, and these clusters are selected as summary candidates. With a user specified summary length, our system generates the corresponding summary by selecting representative frames from candidate clusters and assembling them by their original temporal order. The experimental evaluation demonstrates the effectiveness of our summarization method.
---
paper_title: Automatic generation of video summaries for historical films

paper_content:

A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. In the ECHO (European Chronicles On-line) project, a system was developed to store and manage large collections of historical films for the preservation of cultural heritage. At the University of Mannheim, we have developed the video summarization component of the ECHO system. We discuss the particular challenges the historical film material poses, and how we have designed new video processing algorithms and modified existing ones to cope with noisy black-and-white films.
---
paper_title: VSUM: summarizing from videos

paper_content:

Summarization on produced type of video data (like news or movies) is to find important segments that contain rich information. Users could obtain the important messages by reading summaries rather than full documents. The research in this area could be divided into two parts: (I) image processing (IP) perspective, and (2) NLP (nature language processing) perspective. The former put emphasis on the detection of key frames, while the later focused on the extraction of important concepts. This paper proposes a video summarization system, VSUM. VSUM first identifies all caption words, and then adopts a technique to find the important segments. An external thesaurus is also used in VSUM to enhance the summary extraction process. The experimental results show that VSUM could perform well even if the accuracy of OCR (optical character recognition) is not sophisticating.
---
paper_title: Video summarization and scene detection by graph modeling

paper_content:

We propose a unified approach for video summarization based on the analysis of video structures and video highlights. Two major components in our approach are scene modeling and highlight detection. Scene modeling is achieved by normalized cut algorithm and temporal graph analysis, while highlight detection is accomplished by motion attention modeling. In our proposed approach, a video is represented as a complete undirected graph and the normalized cut algorithm is carried out to globally and optimally partition the graph into video clusters. The resulting clusters form a directed temporal graph and a shortest path algorithm is proposed to efficiently detect video scenes. The attention values are then computed and attached to the scenes, clusters, shots, and subshots in a temporal graph. As a result, the temporal graph can inherently describe the evolution and perceptual importance of a video. In our application, video summaries that emphasize both content balance and perceptual quality can be generated directly from a temporal graph that embeds both the structure and attention information.
---
paper_title: Low cost soccer video summaries based on visual rhythm

paper_content:

The visual rhythm is a spatio-temporal sampled representation of video data providing compact information while preserving several types of video events.We exploit these properties in the present work to propose two new low level descriptors for the analysis of soccer videos computed directly from the visual rhythm.The descriptors are related to dominant color and camera motion estimation.The new descriptors are applied in different tasks aiming the analysis of soccer videos such as shot transition detection, shot classification and attack direction estimation.We also present a simple automated soccer summary application to illustrate the use of the new descriptors.
---
paper_title: MSN: statistical understanding of broadcasted baseball video using multi-level semantic network

paper_content:

The information processing of sports video yields valuable semantics for content delivery over narrowband networks. Traditional image/video processing is formulated in terms of low-level features describing image/video structure and intensity, while the high-level knowledge such as common sense and human perceptual knowledge are encoded in abstract and nongeometric representations. The management of semantic information in video becomes more and more difficult because of the large difference in representations, levels of knowledge, and abstract episodes. This paper proposes a semantic highlight detection scheme using a Multi-level Semantic Network (MSN) for baseball video interpretation. The probabilistic structure can be applied for highlight detection and shot classification. Satisfactory results will be shown to illustrate better performance compared with the traditional ones.
---
paper_title: MINMAX optimal video summarization

paper_content:

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth and/or power are limited. In this paper, our work is based on a MINMAX optimization formulation with viewing time, frame skip and bit rate constraints. New metrics for missing frame and video summary distortions are introduced. Optimal algorithm based on dynamic programming is presented along with experimental results.
---
paper_title: Video skimming and characterization through the combination of image and language understanding

paper_content:

Digital video is rapidly becoming important for education, entertainment and a host of multimedia applications. With the size of the video collections growing to thousands of hours, technology is needed to effectively browse segments in a short time without losing the content of the video. We propose a method to extract the significant audio and video information and create a skim video which represents a very short synopsis of the original. The goal of this work is to show the utility of integrating language and image understanding techniques for video skimming by extraction of significant information, such as specific objects, audio keywords and relevant video structure. The resulting skim video is much shorter; where compaction is as high as 20:1, and yet retains the essential content of the original segment. We have conducted a user-study to test the content summarization and effectiveness of the skim as a browsing tool.
---
paper_title: An innovative algorithm for key frame extraction in video summarization

paper_content:

Video summarization, aimed at reducing the amount of data that must be examined in order to retrieve the information desired from information in a video, is an essential task in video analysis and indexing applications. We propose an innovative approach for the selection of representative (key) frames of a video sequence for video summarization. By analyzing the differences between two consecutive frames of a video sequence, the algorithm determines the complexity of the sequence in terms of changes in the visual content expressed by different frame descriptors. The algorithm, which escapes the complexity of existing methods based, for example, on clustering or optimization strategies, dynamically and rapidly selects a variable number of key frames within each sequence. The key frames are extracted by detecting curvature points within the curve of the cumulative frame differences. Another advantage is that it can extract the key frames on the fly: curvature points can be determined while computing the frame differences and the key frames can be extracted as soon as a second high curvature point has been detected. We compare the performance of this algorithm with that of other key frame extraction algorithms based on different approaches. The summaries obtained have been objectively evaluated by three quality measures: the Fidelity measure, the Shot Reconstruction Degree measure and the Compression Ratio measure.
---
paper_title: Highlight sound effects detection in audio stream

paper_content:

This paper addresses the problem of highlight sound effects detection in audio stream, which is very useful in fields of video summarization and highlight extraction. Unlike researches on audio segmentation and classification, in this domain, it just locates those highlight sound effects in audio stream. An extensible framework is proposed and in current system three sound effects are considered: laughter, applause and cheer, which are tied up with highlight events in entertainments, sports, meetings and home videos. HMMs are used to model these sound effects and a log-likelihood scores based method is used to make final decision. A sound effect attention model is also proposed to extend general audio attention model for highlight extraction and video summarization. Evaluations on a 2-hours audio database showed very encouraging results.
---
paper_title: Classification of self-consumable highlights for soccer video summaries

paper_content:

An effective scheme for soccer summarization is significant to improve the usage of this massively growing video data. The paper presents an extension to our recent work which proposed a framework to integrate highlights into play-breaks to construct more complete soccer summaries. The current focus is to demonstrate the benefits of detecting some specific audio-visual features during play-break sequences in order to classify highlights contained within them. The main purpose is to generate summaries which are self-consumable individually. To support this framework, the algorithms for shot classification and detection of near-goal and slow-motion replay scenes is described. The results of our experiment using 5 soccer videos (20 minutes each) show the performance and reliability of our framework
---
paper_title: Augmented segmentation and visualization for presentation videos

paper_content:

We investigate methods of segmenting, visualizing, and indexing presentation videos by both audio and visual data. The audio track is segmented by speaker, and augmented with key phrases which are extracted using an Automatic Speech Recognizer (ASR). The video track is segmented by visual dissimilarities and changes in speaker gesturing, and augmented by representative key frames. An interactive user interface combines a visual representation of audio, video, text, key frames, and allows the user to navigate presentation videos. User studies with 176 students of varying knowledge were conducted on 7.5 hours of student presentation video (32 presentations). Tasks included searching for various portions of presentations, both known and unknown to students, and summarizing presentations given the annotations. The results are favorable towards the video summaries and the interface, suggesting faster responses by a factor of 20% compared to having access to the actual video. Accuracy of responses remained the same on average. Follow-up surveys present a number of suggestions towards improving the interface, such as the incorporation of automatic speaker clustering and identification, and the display of an abstract topological view of the presentation. Surveys also show alternative contexts in which students would like to use the tool in the classroom environment.
---
paper_title: A fast layout algorithm for visual video summaries

paper_content:

We created an improved layout algorithm for automatically generating visual video summaries reminiscent of comic book pages. The summaries are comprised of images from the video that are sized according to their importance. The algorithm performs a global optimization with respect to a layout cost function that encompasses features such as the number of resized images and the amount of whitespace in the presentation. The algorithm creates summaries that: always fit exactly into the requested area, are varied by containing few rows with images of the same size, and have little whitespace at the end of the last row. The layout algorithm is fast enough to allow the interactive resizing of the summaries and the subsequent generation of a new layout.
---
paper_title: Spatio-temporal quality assessment for home videos

paper_content:

Compared with the video programs taken by professionals, home videos are always with low-quality content resulted from lack of professional capture skills. In this paper, we present a novel spatio-temporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, sub-shot, is selected as the basic unit for quality assessment. A set of spatio-temporal artifacts, regarded as the key factors affecting the overall perceived quality (i.e. unstableness, jerkiness, infidelity, blurring, brightness and orientation), are mined from each sub-shot based on the particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three different methods, including user study, factor fusion, and a learning-based scheme. To validate the proposed scheme, we present a scalable quality-based home video summarization system, aiming at achieving the best quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model based video skimming approach demonstrated the effectiveness of the proposed quality assessment scheme.
---
paper_title: Video summarization by spatial-temporal graph optimization

paper_content:

In this paper we present a novel approach for video summarization based on graph optimization. Our approach emphasizes both a comprehensive visual-temporal content coverage and visual coherence of the video summary. The approach has three stages. First, the source video is segmented into video shots, and a candidate shot set is selected from the video shots according to some video features. Second, a dissimilarity function is defined between the video shots to describe their spatial-temporal relation, and the candidate video shot set is modelled into a directional graph. Third, we outline a dynamic programming algorithm and use it to search the longest path in the graph as the final video skimming. A static video summary is generated at the same time. Experimental results show encouraging promises of our approach for video summarization.
---
paper_title: Affective video content representation and modeling

paper_content:

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.
---
paper_title: Video caption detection and extraction using temporal information

paper_content:

Video caption detection and extraction is an important step for information retrieval in video databases. In this paper, we extract text information in video by fully utilizing the temporal information contained in the video. First we create a binary abstract sequence from a video segment. By analyzing the statistical pixel changes in the sequence, we can effectively locate the (dis)appealing frames of captions. Finally we extract the captions to create a summary of the video segment.
---
paper_title: Automatic Soccer Video Analysis and Summarization

paper_content:

We propose a fully automatic and computationally efficient framework for analysis and summarization of soccer videos using cinematic and object-based features. The proposed framework includes some novel low-level processing algorithms, such as dominant color region detection, robust shot boundary detection, and shot classification, as well as some higher-level algorithms for goal detection, referee detection, and penalty-box detection. The system can output three types of summaries: i) all slow-motion segments in a game; ii) all goals in a game; iii) slow-motion segments classified according to object-based features. The first two types of summaries are based on cinematic features only for speedy processing, while the summaries of the last type contain higher-level semantics. The proposed framework is efficient, effective, and robust. It is efficient in the sense that there is no need to compute object-based features when cinematic features are sufficient for the detection of certain events, e.g., goals in soccer. It is effective in the sense that the framework can also employ object-based features when needed to increase accuracy (at the expense of more computation). The efficiency, effectiveness, and robustness of the proposed framework are demonstrated over a large data set, consisting of more than 13 hours of soccer video, captured in different countries and under different conditions.
---
paper_title: Dynamic storyboards for video content summarization

paper_content:

We propose an innovative, general purpose, approach to the selection and hierarchical representation of key frames of a video sequence for video summarization. In the first stage the shot detection module performs the video structural analysis; in the second stage the key frame extraction module creates the visual summary; and the last stage the summary post-processing module, after a pre-a classification aimed remove meaningless key frames, create a multilevel storyboard that the user may browse.
---
paper_title: Two-Stage Hierarchical Video Summary Extraction to Match Low-Level User Browsing Preferences

paper_content:

A compact summary of video that conveys visual content at various levels of detail enhances user interaction significantly. In this paper, we propose a two-stage framework to generate MPEG-7-compliant hierarchical key frame summaries of video sequences. At the first stage, which is carried out off-line at the time of content production, fuzzy clustering and data pruning methods are applied to given video segments to obtain a nonredundant set of key frames that comprise the finest level of the hierarchical summary. The number of key frames allocated to each shot or segment is determined dynamically and without user supervision through the use of cluster validation techniques. A coarser summary is generated on-demand in the second stage by reducing the number of key frames to match the low-level browsing preferences of a user. The proposed method has been validated by experimental results on a collection of video programs.
---
paper_title: A highlight scene detection and video summarization system using audio feature for a personal video recorder

paper_content:

The personal video recorder such as recordable-DVD recorder, Blu-ray disc recorder and/or hard disc recorder has become popular for a large volume storage device for video/audio content data and a browsing function that would quickly provide a desired scene to the user is required as an essential part of such a large capacity recording/playback system. We propose a highlight scene detection function by using only 'audio' features and realize a browsing function for the recorder that enables completely automatic detection of sports highlights. We detect sports highlights by identifying portions with "commentator's excited speech" using Gaussian mixture models (GMM's) trained using the MDL criterion. Our computation is carried out directly on the MDCT coefficients from the AC-3 coefficients thus giving us a tremendous speed advantage. Our accuracy of detection of sports highlights is high across a variety of sports.
---
paper_title: Flashlight and player detection in fighting sport for video summarization

paper_content:

In this paper, we present a method for generating fighting sport videos summary highlights using flashlight [N. Benjamas et al., May 2005] and player detection by detecting frames that contain close-up flashlight and players. Detected flashlight and distance between players is utilized in efficient summarization of fighting sport videos. The proposed algorithm first player was detected using skin color detection and connected component labeling. Then, the algorithm identifies the correct frame by calculating the distance between players that distance between one player and another one. Our algorithm accurately detects distance between players that it has the ability to capture inherently important events.
---
paper_title: A utility framework for the automatic generation of audio-visual skims

paper_content:

In this paper, we present a novel algorithm for generating audio-visual skims from computable scenes. Skims are useful for browsing digital libraries, and for on-demand summaries in set-top boxes. A computable scene is a chunk of data that exhibits consistencies with respect to chromaticity, lighting and sound. There are three key aspects to our approach: (a) visual complexity and grammar, (b) robust audio segmentation and (c) an utility model for skim generation. We define a measure of visual complexity of a shot, and map complexity to the minimum time for comprehending the shot. Then, we analyze the underlying visual grammar, since it makes the shot sequence meaningful. We segment the audio data into four classes, and then detect significant phrases in the speech segments. The utility functions are defined in terms of complexity and duration of the segment. The target skim is created using a general constrained utility maximization procedure that maximizes the information content and the coherence of the resulting skim. The objective function is constrained due to multimedia synchronization constraints, visual syntax and by penalty functions on audio and video segments. The user study results indicate that the optimal skims show statistically significant differences with other skims with compression rates up to 90%.
---
paper_title: Adaptive extraction of highlights from a sport video based on excitement modeling

paper_content:

This paper addresses the challenge of automatically extracting the highlights from sports TV broadcasts. In particular, we are interested in finding a generic method of highlights extraction, which does not require the development of models for the events that are thought to be interpreted by the users as highlights. Instead, we search for highlights in those video segments that are expected to excite the users most. It is namely realistic to assume that a highlighting event induces a steady increase in a user's excitement, as compared to other, less interesting events. We mimic the expected variations in a user's excitement by observing the temporal behavior of selected audiovisual low-level features and the editing scheme of a video. Relations between this noncontent information and the evoked excitement are drawn partly from psychophysiological research and partly from analyzing the live-video directing practice. The expected variations in a user's excitement are represented by the excitement time curve, which is, subsequently, filtered in an adaptive way to extract the highlights in the prespecified total length and in view of the preferences regarding the highlights strength: extraction can namely be performed with variable sensitivity to capture few "strong" highlights or more "less strong" ones. We evaluate and discuss the performance of our method on the case study of soccer TV broadcasts.
---
paper_title: Semantic units detection and summarization of baseball videos

paper_content:

A framework for analyzing baseball videos and generation of game summary is proposed. Due to the well-defined rules of baseball games, the system efficiently detects semantic units by the domain-related knowledge, and therefore, automatically discovers the structure of a baseball game. After extracting the information changes that are caused by some semantic events on the superimposed caption, a rule-based decision tree is applied to detect meaningful events. Only three types of information, including number-of-outs, score, and base occupation status, are taken in the detection process, and thus the framework detects events and produces summarization in an efficient and effective manner. The experimental results show the effectiveness of this framework and some research opportunities about generating semantic-level summary for sports videos.
---
paper_title: Information theory-based shot cut/fade detection and video summarization

paper_content:

New methods for detecting shot boundaries in video sequences and for extracting key frames using metrics based on information theory are proposed. The method for shot boundary detection relies on the mutual information (MI) and the joint entropy (JE) between the frames. It can detect cuts, fade-ins and fade-outs. The detection technique was tested on the TRECVID2003 video test set having different types of shots and containing significant object and camera motion inside the shots. It is demonstrated that the method detects both fades and abrupt cuts with high accuracy. The information theory measure provides us with better results because it exploits the inter-frame information in a more compact way than frame subtraction. It was also successfully compared to other methods published in literature. The method for key frame extraction uses MI as well. We show that it captures satisfactorily the visual content of the shot.
---
paper_title: Automatically generating summaries for musical video

paper_content:

In this paper, we propose a novel approach to automatically summarize musical videos. The proposed summarization scheme is different from the current methods used for video summarization. The musical video is separated into the musical and visual tracks. A music summary is created by analyzing the music content based on music features, adaptive clustering algorithm and musical domain knowledge. Then, shots are detected and clustered in the visual track. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of summarization. The experiments on different genres of musical video and comparisons with the summaries only based on music track and video track indicate that the results of summarization using proposed method are significant and effective to help realize user's expectation.
---
paper_title: An approach to generating two-level video abstraction

paper_content:

Video abstraction is a short summary of the content of a longer video document. Most existing video abstraction methods are based on shot-level, which is not sufficient to meaningful browsing and is too fine to users sometimes. In this paper, we propose a novel approach of generating video abstraction at two levels, namely, the shot-level and the scene-level. We put up a method of extracting key frames from shots, according to the content variation of the latter. An updated time-adaptive algorithm is used to group the shots into scene and representative frames are extracted in the region of that scene using the method of generating Minimum Spanning Tree. Key frames and representative frames can represent the content of shots and scenes, respectively. The organized sequences of key frames and representative frames are the two-level video abstraction. Experiments based on real-world movies show that the method above can provide users with better video summary at different levels.
---
paper_title: Summarization of news video and its description for content‐based access

paper_content:

A video summary abstracts the entirety with the gist without losing the essential content of the original video and also facilitates efficient content-based access to the desired content. In this article, we propose a novel method for summarizing a news video based on multimodal analysis of the content. The proposed method exploits the closed caption (CC) data to locate semantically meaningful highlights in a news video and speech signals in an audio stream to align the CC data with the video in a time-line. Then, the extracted highlights are described in a multilevel structure using the MPEG-7 Summarization Description Scheme (DS). Specifically, we use the HierarchicalSummary DS that allows efficient accessing of the content through such functionalities as multilevel abstracts and navigation guidance in a hierarchical fashion. Intensive experiments with our prototypical systems are presented to demonstrate the validity and reliability of the proposed method in real applications. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13, 267–274, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10067
---
paper_title: Narrative abstraction model for story-oriented video

paper_content:

TV program review services, especially drama review services, are one of the most popular video on demand services on the Web. In this paper, we propose a novel video abstraction model for a review service of story-oriented video such as dramas. In a drama review service, viewers want to understand the story in a short time and service providers want to provide video abstracts at minimum cost. The proposed model enables the automatic creation of a video abstract that still allows viewers to understand the overall story of the source video. Also, the model has a flexible structure so that the duration of an abstract can be adjusted depending on the requirements given by viewers. We get clues for human understanding of a story from scenario writing rules and editorial techniques which are popularly used in the process of video producing. We have implemented the proposed model and successfully applied it to several TV dramas.
---
paper_title: Automatically extracting highlights for TV Baseball programs

paper_content:

In today's fast-paced world, while the number of channels of television programming available is increasing rapidly, the time available to watch them remains the same or is decreasing. Users desire the capability to watch the programs time-shifted (on-demand) and/or to watch just the highlights to save time. In this paper we explore how to provide for the latter capability, that is the ability to extract highlights automatically, so that viewing time can be reduced. We focus on the sport of baseball as our initial target—it is a very popular sport, the whole game is quite long, and the exciting portions are few. We focus on detecting highlights using audio-track features alone without relying on expensive-to-compute video-track features. We use a combination of generic sports features and baseball-specific features to obtain our results, but believe that may other sports offer the same opportunity and that the techniques presented here will apply to those sports. We present details on relative performance of various learning algorithms, and a probabilistic framework for combining multiple sources of information. We present results comparing output of our algorithms against human-selected highlights for a diverse collection of baseball games with very encouraging results.
---
paper_title: Sequential association mining for video summarization

paper_content:

In this paper, we propose an association-based video summarization scheme that mines sequential associations from video data for summary creation. Given detected shots of video V, we first cluster them into visually distinct groups, and then construct a sequential sequence by integrating the temporal order and cluster type of each shot. An association mining scheme is designed to mine sequentially associated clusters from the sequence, and these clusters are selected as summary candidates. With a user specified summary length, our system generates the corresponding summary by selecting representative frames from candidate clusters and assembling them by their original temporal order. The experimental evaluation demonstrates the effectiveness of our summarization method.
---
paper_title: Automatic generation of video summaries for historical films

paper_content:

A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. In the ECHO (European Chronicles On-line) project, a system was developed to store and manage large collections of historical films for the preservation of cultural heritage. At the University of Mannheim, we have developed the video summarization component of the ECHO system. We discuss the particular challenges the historical film material poses, and how we have designed new video processing algorithms and modified existing ones to cope with noisy black-and-white films.
---
paper_title: VSUM: summarizing from videos

paper_content:

Summarization on produced type of video data (like news or movies) is to find important segments that contain rich information. Users could obtain the important messages by reading summaries rather than full documents. The research in this area could be divided into two parts: (I) image processing (IP) perspective, and (2) NLP (nature language processing) perspective. The former put emphasis on the detection of key frames, while the later focused on the extraction of important concepts. This paper proposes a video summarization system, VSUM. VSUM first identifies all caption words, and then adopts a technique to find the important segments. An external thesaurus is also used in VSUM to enhance the summary extraction process. The experimental results show that VSUM could perform well even if the accuracy of OCR (optical character recognition) is not sophisticating.
---
paper_title: Video summarization and scene detection by graph modeling

paper_content:

We propose a unified approach for video summarization based on the analysis of video structures and video highlights. Two major components in our approach are scene modeling and highlight detection. Scene modeling is achieved by normalized cut algorithm and temporal graph analysis, while highlight detection is accomplished by motion attention modeling. In our proposed approach, a video is represented as a complete undirected graph and the normalized cut algorithm is carried out to globally and optimally partition the graph into video clusters. The resulting clusters form a directed temporal graph and a shortest path algorithm is proposed to efficiently detect video scenes. The attention values are then computed and attached to the scenes, clusters, shots, and subshots in a temporal graph. As a result, the temporal graph can inherently describe the evolution and perceptual importance of a video. In our application, video summaries that emphasize both content balance and perceptual quality can be generated directly from a temporal graph that embeds both the structure and attention information.
---
paper_title: Low cost soccer video summaries based on visual rhythm

paper_content:

The visual rhythm is a spatio-temporal sampled representation of video data providing compact information while preserving several types of video events.We exploit these properties in the present work to propose two new low level descriptors for the analysis of soccer videos computed directly from the visual rhythm.The descriptors are related to dominant color and camera motion estimation.The new descriptors are applied in different tasks aiming the analysis of soccer videos such as shot transition detection, shot classification and attack direction estimation.We also present a simple automated soccer summary application to illustrate the use of the new descriptors.
---
paper_title: MSN: statistical understanding of broadcasted baseball video using multi-level semantic network

paper_content:

The information processing of sports video yields valuable semantics for content delivery over narrowband networks. Traditional image/video processing is formulated in terms of low-level features describing image/video structure and intensity, while the high-level knowledge such as common sense and human perceptual knowledge are encoded in abstract and nongeometric representations. The management of semantic information in video becomes more and more difficult because of the large difference in representations, levels of knowledge, and abstract episodes. This paper proposes a semantic highlight detection scheme using a Multi-level Semantic Network (MSN) for baseball video interpretation. The probabilistic structure can be applied for highlight detection and shot classification. Satisfactory results will be shown to illustrate better performance compared with the traditional ones.
---
paper_title: MINMAX optimal video summarization

paper_content:

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth and/or power are limited. In this paper, our work is based on a MINMAX optimization formulation with viewing time, frame skip and bit rate constraints. New metrics for missing frame and video summary distortions are introduced. Optimal algorithm based on dynamic programming is presented along with experimental results.
---
paper_title: Video skimming and characterization through the combination of image and language understanding

paper_content:

Digital video is rapidly becoming important for education, entertainment and a host of multimedia applications. With the size of the video collections growing to thousands of hours, technology is needed to effectively browse segments in a short time without losing the content of the video. We propose a method to extract the significant audio and video information and create a skim video which represents a very short synopsis of the original. The goal of this work is to show the utility of integrating language and image understanding techniques for video skimming by extraction of significant information, such as specific objects, audio keywords and relevant video structure. The resulting skim video is much shorter; where compaction is as high as 20:1, and yet retains the essential content of the original segment. We have conducted a user-study to test the content summarization and effectiveness of the skim as a browsing tool.
---
paper_title: An innovative algorithm for key frame extraction in video summarization

paper_content:

Video summarization, aimed at reducing the amount of data that must be examined in order to retrieve the information desired from information in a video, is an essential task in video analysis and indexing applications. We propose an innovative approach for the selection of representative (key) frames of a video sequence for video summarization. By analyzing the differences between two consecutive frames of a video sequence, the algorithm determines the complexity of the sequence in terms of changes in the visual content expressed by different frame descriptors. The algorithm, which escapes the complexity of existing methods based, for example, on clustering or optimization strategies, dynamically and rapidly selects a variable number of key frames within each sequence. The key frames are extracted by detecting curvature points within the curve of the cumulative frame differences. Another advantage is that it can extract the key frames on the fly: curvature points can be determined while computing the frame differences and the key frames can be extracted as soon as a second high curvature point has been detected. We compare the performance of this algorithm with that of other key frame extraction algorithms based on different approaches. The summaries obtained have been objectively evaluated by three quality measures: the Fidelity measure, the Shot Reconstruction Degree measure and the Compression Ratio measure.
---
paper_title: Optimizing user expectations for video semantic filtering and abstraction

paper_content:

We describe a novel automatic system that generates personalized videos based on semantic filtering or summarization techniques. This system uses a new set of more than one hundred visual semantic detectors that automatically detect video concepts in faster than realtime. Based on personal profiles, the system generates either video summaries from video databases or filtered video contents from live broadcasting videos. The prototype experiments have shown the effectiveness and stabilities of the system.
---
paper_title: Highlight sound effects detection in audio stream

paper_content:

This paper addresses the problem of highlight sound effects detection in audio stream, which is very useful in fields of video summarization and highlight extraction. Unlike researches on audio segmentation and classification, in this domain, it just locates those highlight sound effects in audio stream. An extensible framework is proposed and in current system three sound effects are considered: laughter, applause and cheer, which are tied up with highlight events in entertainments, sports, meetings and home videos. HMMs are used to model these sound effects and a log-likelihood scores based method is used to make final decision. A sound effect attention model is also proposed to extend general audio attention model for highlight extraction and video summarization. Evaluations on a 2-hours audio database showed very encouraging results.
---
paper_title: Semantic video summarization in compressed domain MPEG video

paper_content:

In this paper, we present a semantic summarization algorithm that interfaces with the metadata and that works in compressed domain, in particular MPEG-1 and MPEG-2 videos. In enabling a summarization algorithm through high-level semantic content, we try to address two major problems. First, we present the facility provided in the DVA system that allows the semi-automatic creation of this metadata. Second, we address the main point of this system which is the utilization of this metadata to filter out frames, creating an abstract of a video summary quality survey indicates that the proposed method performs satisfactorily.
---
paper_title: Personalized abstraction of broadcasted American football video by highlight selection

paper_content:

Video abstraction is defined as creating shorter video clips or video posters from an original video stream. In this paper, we propose a method of generating a personalized abstract of broadcasted American football video. We first detect significant events in the video stream by matching textual overlays appearing in an image frame with the descriptions of gamestats in which highlights of the game are described. Then, we select highlight shots which should be included in the video abstract from those detected events reflecting on their significance degree and personal preferences, and generate a video clip by connecting the shots augmented with related audio and text. An hour-length video can be compressed into a minute-length personalized abstract. We experimentally verified the effectiveness of this method by comparing man-made video abstracts.
---
paper_title: Content-based multimedia information retrieval: State of the art and challenges

paper_content:

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100p recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.
---
paper_title: Classification of self-consumable highlights for soccer video summaries

paper_content:

An effective scheme for soccer summarization is significant to improve the usage of this massively growing video data. The paper presents an extension to our recent work which proposed a framework to integrate highlights into play-breaks to construct more complete soccer summaries. The current focus is to demonstrate the benefits of detecting some specific audio-visual features during play-break sequences in order to classify highlights contained within them. The main purpose is to generate summaries which are self-consumable individually. To support this framework, the algorithms for shot classification and detection of near-goal and slow-motion replay scenes is described. The results of our experiment using 5 soccer videos (20 minutes each) show the performance and reliability of our framework
---
paper_title: Automatic video summarizing tool using MPEG-7 descriptors for personal video recorder

paper_content:

We introduce an automatic video summarizing tool (AVST) for a personal video recorder. The tool utilizes MPEG-7 visual descriptors to generate a video index for a summary. The resulting index generates not only a preview of a movie but also allows non-linear access with thumbnails. In addition, the index supports the searching of shots similar to a desired one within saved video sequences. Moreover, simple shot-based video editing can readily be achieved using the generated index.
---
paper_title: Personalized video summary using visual semantic annotations and automatic speech transcriptions

paper_content:

A personalized video summary is dynamically generated in our video personalization and summary system based on user preference and usage environment. The three-tier personalization system adopts the server-middleware-client architecture in order maintain, select, adapt, and deliver rich media content to the user. The server stores the content sources along with their corresponding MPEG-7 metadata descriptions. In this paper, the metadata includes visual semantic annotations and automatic speech transcriptions. Our personalization and summarization engine in the middleware selects the optimal set of desired video segments by matching shot annotations and sentence transcripts with user preferences. The process includes the shot-to-sentence alignment, summary segment selection, and user preference matching and propagation. As a result, the relevant visual shot and audio sentence segments are aggregated and composed into a personalized video summary.
---
paper_title: Media content and type selection from always-on wearable video

paper_content:

A system is described for summarizing head-mounted or hand-carried "always-on" video. The example used is a tourist walking around a historic city with friends and family. The summary consists of a mixture of stills, panoramas and video clips. The system identifies both the scenes to appear in the summary and the media type used to represent them. As there are few shot boundaries in this class of video, the decisions are based on the system's classification of the user's behaviour demonstrated by the motion of the camera, and motion in the scene.
---
paper_title: Video summarization based on user log enhanced link analysis

paper_content:

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
---
paper_title: A fast layout algorithm for visual video summaries

paper_content:

We created an improved layout algorithm for automatically generating visual video summaries reminiscent of comic book pages. The summaries are comprised of images from the video that are sized according to their importance. The algorithm performs a global optimization with respect to a layout cost function that encompasses features such as the number of resized images and the amount of whitespace in the presentation. The algorithm creates summaries that: always fit exactly into the requested area, are varied by containing few rows with images of the same size, and have little whitespace at the end of the last row. The layout algorithm is fast enough to allow the interactive resizing of the summaries and the subsequent generation of a new layout.
---
paper_title: Tennis video abstraction from audio and visual cues

paper_content:

We propose a context-based model of video abstraction exploiting both audio and video features and applied to tennis TV programs. We can automatically produce different types of summary of a given video depending on the users' constraints or preferences. We have first designed an efficient and accurate temporal segmentation of the video into segments homogeneous w.r.t the camera motion. We introduce original visual descriptors related to the dominant and residual image motions. The different summary types are obtained by specifying adapted classification criteria which involve audio features to select the relevant segments to be included in the video abstract. The proposed scheme has been validated on 22 hours of tennis videos.
---
paper_title: Spatio-temporal quality assessment for home videos

paper_content:

Compared with the video programs taken by professionals, home videos are always with low-quality content resulted from lack of professional capture skills. In this paper, we present a novel spatio-temporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, sub-shot, is selected as the basic unit for quality assessment. A set of spatio-temporal artifacts, regarded as the key factors affecting the overall perceived quality (i.e. unstableness, jerkiness, infidelity, blurring, brightness and orientation), are mined from each sub-shot based on the particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three different methods, including user study, factor fusion, and a learning-based scheme. To validate the proposed scheme, we present a scalable quality-based home video summarization system, aiming at achieving the best quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model based video skimming approach demonstrated the effectiveness of the proposed quality assessment scheme.
---
paper_title: Video summarization by spatial-temporal graph optimization

paper_content:

In this paper we present a novel approach for video summarization based on graph optimization. Our approach emphasizes both a comprehensive visual-temporal content coverage and visual coherence of the video summary. The approach has three stages. First, the source video is segmented into video shots, and a candidate shot set is selected from the video shots according to some video features. Second, a dissimilarity function is defined between the video shots to describe their spatial-temporal relation, and the candidate video shot set is modelled into a directional graph. Third, we outline a dynamic programming algorithm and use it to search the longest path in the graph as the final video skimming. A static video summary is generated at the same time. Experimental results show encouraging promises of our approach for video summarization.
---
paper_title: Affective video content representation and modeling

paper_content:

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.
---
paper_title: Video caption detection and extraction using temporal information

paper_content:

Video caption detection and extraction is an important step for information retrieval in video databases. In this paper, we extract text information in video by fully utilizing the temporal information contained in the video. First we create a binary abstract sequence from a video segment. By analyzing the statistical pixel changes in the sequence, we can effectively locate the (dis)appealing frames of captions. Finally we extract the captions to create a summary of the video segment.
---
paper_title: Automatic Soccer Video Analysis and Summarization

paper_content:

We propose a fully automatic and computationally efficient framework for analysis and summarization of soccer videos using cinematic and object-based features. The proposed framework includes some novel low-level processing algorithms, such as dominant color region detection, robust shot boundary detection, and shot classification, as well as some higher-level algorithms for goal detection, referee detection, and penalty-box detection. The system can output three types of summaries: i) all slow-motion segments in a game; ii) all goals in a game; iii) slow-motion segments classified according to object-based features. The first two types of summaries are based on cinematic features only for speedy processing, while the summaries of the last type contain higher-level semantics. The proposed framework is efficient, effective, and robust. It is efficient in the sense that there is no need to compute object-based features when cinematic features are sufficient for the detection of certain events, e.g., goals in soccer. It is effective in the sense that the framework can also employ object-based features when needed to increase accuracy (at the expense of more computation). The efficiency, effectiveness, and robustness of the proposed framework are demonstrated over a large data set, consisting of more than 13 hours of soccer video, captured in different countries and under different conditions.
---
paper_title: Dynamic storyboards for video content summarization

paper_content:

We propose an innovative, general purpose, approach to the selection and hierarchical representation of key frames of a video sequence for video summarization. In the first stage the shot detection module performs the video structural analysis; in the second stage the key frame extraction module creates the visual summary; and the last stage the summary post-processing module, after a pre-a classification aimed remove meaningless key frames, create a multilevel storyboard that the user may browse.
---
paper_title: Two-Stage Hierarchical Video Summary Extraction to Match Low-Level User Browsing Preferences

paper_content:

A compact summary of video that conveys visual content at various levels of detail enhances user interaction significantly. In this paper, we propose a two-stage framework to generate MPEG-7-compliant hierarchical key frame summaries of video sequences. At the first stage, which is carried out off-line at the time of content production, fuzzy clustering and data pruning methods are applied to given video segments to obtain a nonredundant set of key frames that comprise the finest level of the hierarchical summary. The number of key frames allocated to each shot or segment is determined dynamically and without user supervision through the use of cluster validation techniques. A coarser summary is generated on-demand in the second stage by reducing the number of key frames to match the low-level browsing preferences of a user. The proposed method has been validated by experimental results on a collection of video programs.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Design and evaluation of a music video summarization system

paper_content:

We present a system that summarizes the textual, audio, and video information of music videos in a format tuned to the preferences of a focus group of 20 users. First, we analyzed user-needs for the content and the layout of the music summaries. Then, we designed algorithms that segment individual song videos from full music video programs by noting changes in color palette, transcript, and audio classification. We summarize each song with automatically selected high level information such as title, artist, duration, title frame, and text as well as audio and visual segments of the chorus. Our system automatically determines with high recall and precision chorus locations, from the placement of repeated words and phrases in the text of the song's lyrics. Our Bayesian belief network then selects other significant video and audio content from the multiple media. Overall, we are able to compress content by a factor of 10. Our second user study has identified the principal variations between users in their choices of content desired in the summary, and in their choices of the platforms that should support their viewing.
---
paper_title: Auto-summarization of audio-video presentations

paper_content:

As streaming audio-video technology becomes widespread, there is a dramatic increase in the amount of multimedia content available on the net. Users face a new challenge: How to examine large amounts of multimedia content quickly. One technique that can enable quick overview of multimedia is video summaries; that is, a shorter version assembled by picking important segments from the original. We evaluate three techniques for automatic creation of summaries for online audio-video presentations. These techniques exploit information in the audio signal (e.g., pitch and pause information), knowledge of slide transition points in the presentation, and information about access patterns of previous users. We report a user study that compares automatically generated summaries that are 20%-25% the length of full presentations to author generated summaries. Users learn from the computer-generated summaries, although less than from authors' summaries. They initially find computer-generated summaries less coherent, but quickly grow accustomed to them.
---
paper_title: A highlight scene detection and video summarization system using audio feature for a personal video recorder

paper_content:

The personal video recorder such as recordable-DVD recorder, Blu-ray disc recorder and/or hard disc recorder has become popular for a large volume storage device for video/audio content data and a browsing function that would quickly provide a desired scene to the user is required as an essential part of such a large capacity recording/playback system. We propose a highlight scene detection function by using only 'audio' features and realize a browsing function for the recorder that enables completely automatic detection of sports highlights. We detect sports highlights by identifying portions with "commentator's excited speech" using Gaussian mixture models (GMM's) trained using the MDL criterion. Our computation is carried out directly on the MDCT coefficients from the AC-3 coefficients thus giving us a tremendous speed advantage. Our accuracy of detection of sports highlights is high across a variety of sports.
---
paper_title: Flashlight and player detection in fighting sport for video summarization

paper_content:

In this paper, we present a method for generating fighting sport videos summary highlights using flashlight [N. Benjamas et al., May 2005] and player detection by detecting frames that contain close-up flashlight and players. Detected flashlight and distance between players is utilized in efficient summarization of fighting sport videos. The proposed algorithm first player was detected using skin color detection and connected component labeling. Then, the algorithm identifies the correct frame by calculating the distance between players that distance between one player and another one. Our algorithm accurately detects distance between players that it has the ability to capture inherently important events.
---
paper_title: A utility framework for the automatic generation of audio-visual skims

paper_content:

In this paper, we present a novel algorithm for generating audio-visual skims from computable scenes. Skims are useful for browsing digital libraries, and for on-demand summaries in set-top boxes. A computable scene is a chunk of data that exhibits consistencies with respect to chromaticity, lighting and sound. There are three key aspects to our approach: (a) visual complexity and grammar, (b) robust audio segmentation and (c) an utility model for skim generation. We define a measure of visual complexity of a shot, and map complexity to the minimum time for comprehending the shot. Then, we analyze the underlying visual grammar, since it makes the shot sequence meaningful. We segment the audio data into four classes, and then detect significant phrases in the speech segments. The utility functions are defined in terms of complexity and duration of the segment. The target skim is created using a general constrained utility maximization procedure that maximizes the information content and the coherence of the resulting skim. The objective function is constrained due to multimedia synchronization constraints, visual syntax and by penalty functions on audio and video segments. The user study results indicate that the optimal skims show statistically significant differences with other skims with compression rates up to 90%.
---
paper_title: Adaptive extraction of highlights from a sport video based on excitement modeling

paper_content:

This paper addresses the challenge of automatically extracting the highlights from sports TV broadcasts. In particular, we are interested in finding a generic method of highlights extraction, which does not require the development of models for the events that are thought to be interpreted by the users as highlights. Instead, we search for highlights in those video segments that are expected to excite the users most. It is namely realistic to assume that a highlighting event induces a steady increase in a user's excitement, as compared to other, less interesting events. We mimic the expected variations in a user's excitement by observing the temporal behavior of selected audiovisual low-level features and the editing scheme of a video. Relations between this noncontent information and the evoked excitement are drawn partly from psychophysiological research and partly from analyzing the live-video directing practice. The expected variations in a user's excitement are represented by the excitement time curve, which is, subsequently, filtered in an adaptive way to extract the highlights in the prespecified total length and in view of the preferences regarding the highlights strength: extraction can namely be performed with variable sensitivity to capture few "strong" highlights or more "less strong" ones. We evaluate and discuss the performance of our method on the case study of soccer TV broadcasts.
---
paper_title: Semantic units detection and summarization of baseball videos

paper_content:

A framework for analyzing baseball videos and generation of game summary is proposed. Due to the well-defined rules of baseball games, the system efficiently detects semantic units by the domain-related knowledge, and therefore, automatically discovers the structure of a baseball game. After extracting the information changes that are caused by some semantic events on the superimposed caption, a rule-based decision tree is applied to detect meaningful events. Only three types of information, including number-of-outs, score, and base occupation status, are taken in the detection process, and thus the framework detects events and produces summarization in an efficient and effective manner. The experimental results show the effectiveness of this framework and some research opportunities about generating semantic-level summary for sports videos.
---
paper_title: Summarizing wearable video

paper_content:

"We want to record our entire life by video" is the motivation of this research. Developing wearable devices and huge storage devices will make it possible to keep entire life by video. We could capture 70 years of our life, however, the problem is how to handle such a huge amount of data. Automatic summarization based on personal interest should be required. In this paper we propose an approach to the automatic structuring and summarization of wearable video. (Wearable video is our abbreviation of "video captured by a wearable camera".) In our approach, we make use of a wearable camera and a sensor of brain waves. The video is firstly structured by objective features of video, and the shots are rated by subjective measures based on brain waves. The approach is very successful for real world experiments and it automatically extracted all the events that the subjects reported they had felt interesting.
---
paper_title: Information theory-based shot cut/fade detection and video summarization

paper_content:

New methods for detecting shot boundaries in video sequences and for extracting key frames using metrics based on information theory are proposed. The method for shot boundary detection relies on the mutual information (MI) and the joint entropy (JE) between the frames. It can detect cuts, fade-ins and fade-outs. The detection technique was tested on the TRECVID2003 video test set having different types of shots and containing significant object and camera motion inside the shots. It is demonstrated that the method detects both fades and abrupt cuts with high accuracy. The information theory measure provides us with better results because it exploits the inter-frame information in a more compact way than frame subtraction. It was also successfully compared to other methods published in literature. The method for key frame extraction uses MI as well. We show that it captures satisfactorily the visual content of the shot.
---
paper_title: Creating audio keywords for event detection in soccer video

paper_content:

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.
---
paper_title: Hierarchical video summarization based on context clustering

paper_content:

A personalized video summary is dynamically generated in our video personalization and summarization system based on user preference and usage environment. The three-tier personalization system adopts the server-middleware-client architecture in order to maintain, select, adapt, and deliver rich media content to the user. The server stores the content sources along with their corresponding MPEG-7 metadata descriptions. In this paper, the metadata includes visual semantic annotations and automatic speech transcriptions. Our personalization and summarization engine in the middleware selects the optimal set of desired video segments by matching shot annotations and sentence transcripts with user preferences. Besides finding the desired contents, the objective is to present a coherent summary. There are diverse methods for creating summaries, and we focus on the challenges of generating a hierarchical video summary based on context information. In our summarization algorithm, three inputs are used to generate the hierarchical video summary output. These inputs are (1) MPEG-7 metadata descriptions of the contents in the server, (2) user preference and usage environment declarations from the user client, and (3) context information including MPEG-7 controlled term list and classification scheme. In a video sequence, descriptions and relevance scores are assigned to each shot. Based on these shot descriptions, context clustering is performed to collect consecutively similar shots to correspond to hierarchical scene representations. The context clustering is based on the available context information, and may be derived from domain knowledge or rules engines. Finally, the selection of structured video segments to generate the hierarchical summary efficiently balances between scene representation and shot selection.
---
paper_title: Dynamic video summarization of home video

paper_content:

An increasing number of people own and use camcorders to make videos that capture their experiences and document their lives. These videos easily add up to many hours of material. Oddly, most of them are put into a storage box and never touched or watched again. The reasons for this are manifold. Firstly, the raw video material is unedited, and is therefore long-winded and lacking visually appealing effects. Video editing would help, but, it is still too time-consuming; people rarely find the time to do it. Secondly, watching the same tape more than a few times can be boring, since the video lacks any variation or surprise during playback. Automatic video abstracting algorithms can provide a method for processing videos so that users will want to play the material more often. However, existing automatic abstracting algorithms have been designed for feature films, newscasts or documentaries, and thus are inappropriate for home video material and raw video footage in general. In this paper, we present new algorithms for generating amusing, visually appealing and variable video abstracts of home video material automatically. They make use of a new, empirically motivated approach, also presented in the paper, to cluster time-stamped shots hierarchically into meaningful units. Last but not least, we propose a simple and natural extension of the way people acquire video - so-called on-the-fly annotations - which will allow a completely new set of applications on raw video footage as well as enable better and more selective automatic video abstracts. Moreover, our algorithms are not restricted to home video but can also be applied to raw video footage in general.
---
paper_title: Efficient retrieval of life log based on context and content

paper_content:

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.
---
paper_title: VideoQA: question answering on news video

paper_content:

When querying a news video archive, the users are interested in retrieving precise answers in the form of a summary that best answers the query. However, current video retrieval systems, including the search engines on the web, are designed to retrieve documents instead of precise answers. This research explores the use of question answering (QA) techniques to support personalized news video retrieval. Users interact with our system, VideoQA, using short natural language questions with implicit constraints on contents, context, duration, and genre of expected videos. VideoQA returns short precise news video summaries as answers. The main contributions of this research are: (a) the extension of QA technology to support QA in news video; and (b) the use of multi-modal features, including visual, audio, textual, and external resources, to help correct speech recognition errors and to perform precise question answering. The system has been tested on 7 days of news video and has been found to be effective.
---
paper_title: Event detection in baseball video using superimposed caption recognition

paper_content:

We have developed a novel system for baseball video event detection and summarization using superimposed caption text detection and recognition. The system detects different types of semantic level events in baseball video including scoring and last pitch of each batter. The system has two components: event detection and event boundary detection. Event detection is realized by change detection and recognition of game stat texts (such as text information showing in score box). Event boundary detection is achieved using our previously developed algorithm, which detects the pitch view as the event beginning and nonactive view as potential endings of the event. One unique contribution of the system is its capability to accurately detect the semantic level events by combining video text recognition with camera view recognition. Another unique feature is the real-time processing speed by taking advantage of compressed-domain approaches in part of the algorithms such as caption detection. To the best of our knowledge, this is the first system achieving accurate detection of multiple types of high-level semantic events in baseball videos.
---
paper_title: An approach to generating two-level video abstraction

paper_content:

Video abstraction is a short summary of the content of a longer video document. Most existing video abstraction methods are based on shot-level, which is not sufficient to meaningful browsing and is too fine to users sometimes. In this paper, we propose a novel approach of generating video abstraction at two levels, namely, the shot-level and the scene-level. We put up a method of extracting key frames from shots, according to the content variation of the latter. An updated time-adaptive algorithm is used to group the shots into scene and representative frames are extracted in the region of that scene using the method of generating Minimum Spanning Tree. Key frames and representative frames can represent the content of shots and scenes, respectively. The organized sequences of key frames and representative frames are the two-level video abstraction. Experiments based on real-world movies show that the method above can provide users with better video summary at different levels.
---
paper_title: Using MPEG-7 and MPEG-21 for personalizing video

paper_content:

As multimedia content has proliferated over the past several years, users have begun to expect that content be easily accessed according to their own preferences. One of the most effective ways to do this is through using the MPEG-7 and MPEG-21 standards, which can help address the issues associated with designing a video personalization and summarization system in heterogeneous usage environments. This three-tier architecture provides a standards-compliant infrastructure that, in conjunction with our tools, can help select, adapt, and deliver personalized video summaries to users. In extending our summarization research, we plan to explore semantic similarities across multiple simultaneous news media sources and to abstract summaries for different viewpoints. Doing so will allow us to track a semantic topic as it evolves into the future. As a result, we should be able to summarize news repositories into a smaller collection of topic threads.
---
paper_title: The CPR model for summarizing video

paper_content:

Most past work on video summarization has been based on selecting key frames from videos. We propose a model of video summarization based on three important parameters: Priority (of frames), Continuity (of the summary), and non-Repetition (of the summary). In short, a summary must include high priority frames, must be continuous and non-repetitive. An optimal summary is one that maximizes an objective function based on these three parameters. We develop formal definitions of all these concepts and provide algorithms to find optimal summaries. We briefly report on the performance of these algorithms.
---
paper_title: Framework for personalized multimedia summarization

paper_content:

We extend and validate methods of personalization to the domain of automatically created multimedia summaries. Based on a previously performed user study of 59 people we derived a mapping of personality profile information to preferred multimedia features. This article describes our summarization algorithm. We define constraints for automatic summary generation. Summaries should consist of contiguous segments of full shots, with duration proportional to the log of video length, selected by an objective function of total "importance" of features, with heuristic rules for deciding the "best" combination of length and importance. We validated the summaries with a user study of 32 people. They were asked to answer a shortened series of personality queries. Using this current user profile, together with the earlier genre-specific reduced mapping and with automatically derived features, we automatically generated two summaries for each video: one optimally matched, and one matched to the "opposite" personality. Each user evaluated both summaries on a preference scale for four each of: news, talk show, and music videos. From a statistical analysis we find statistically significant evidence of the effectiveness of personalization on news and music videos, with no evidence of user subpopulations. We conclude for these genres that our claim, of a universal mapping from certain measured personality traits to the computable creation of preferred multimedia summaries, is supported.
---
paper_title: Efficient access to video content in a unified framework

paper_content:

Book contents' browsing and retrieval have been greatly facilitated by its Table-of-Contents (ToC) and Index, respectively. Unfortunately, today's video lacks such powerful mechanisms. In this paper, we explore and present novel techniques for constructing video ToC and Index. Furthermore, we explore the relationship between video browsing and retrieval and propose a unified framework: to incorporate both entities in a seamless way. Experimental results on real-world video clips justify our proposed framework for providing efficient access to video content.
---
paper_title: Video Summarization for Large Sports Video Archives

paper_content:

Video summarization is defined as creating a shorter video clip or a video poster which includes only the important scenes in the original video streams. In this paper, we propose two methods of generating a summary of arbitrary length for large sports video archives. One is to create a concise video clip by temporally compressing the amount of the video data. The other is to provide a video poster by spatially presenting the image keyframes which together represent the whole video content. Our methods deal with the metadata which has semantic descriptions of video content. Summaries are created according to the significance of each video segment which is normalized in order to handle large sports video archives. We experimentally verified the effectiveness of our methods by comparing the results with man-made video summaries
---
paper_title: The priority curve algorithm for video summarization

paper_content:

In this paper, we introduce the concept of a priority curve associated with a video. We then provide an algorithm that can use the priority curve to create a summary (of a desired length) of any video. The summary thus created exhibits nice continuity properties and also avoids repetition. We have implemented the priority curve algorithm (PCA) and compared it with other summarization algorithms in the literature. We show that PCA is faster than existing algorithms and also produces better quality summaries. The quality of summaries was evaluated by a group of 200 students in Naples, Italy, who watched soccer videos. We also briefly describe a soccer video summarization system we have built on using the PCA architecture and various (classical) image processing algorithms.
---
paper_title: Automatically extracting highlights for TV Baseball programs

paper_content:

In today's fast-paced world, while the number of channels of television programming available is increasing rapidly, the time available to watch them remains the same or is decreasing. Users desire the capability to watch the programs time-shifted (on-demand) and/or to watch just the highlights to save time. In this paper we explore how to provide for the latter capability, that is the ability to extract highlights automatically, so that viewing time can be reduced. We focus on the sport of baseball as our initial target—it is a very popular sport, the whole game is quite long, and the exciting portions are few. We focus on detecting highlights using audio-track features alone without relying on expensive-to-compute video-track features. We use a combination of generic sports features and baseball-specific features to obtain our results, but believe that may other sports offer the same opportunity and that the techniques presented here will apply to those sports. We present details on relative performance of various learning algorithms, and a probabilistic framework for combining multiple sources of information. We present results comparing output of our algorithms against human-selected highlights for a diverse collection of baseball games with very encouraging results.
---
paper_title: Sequential association mining for video summarization

paper_content:

In this paper, we propose an association-based video summarization scheme that mines sequential associations from video data for summary creation. Given detected shots of video V, we first cluster them into visually distinct groups, and then construct a sequential sequence by integrating the temporal order and cluster type of each shot. An association mining scheme is designed to mine sequentially associated clusters from the sequence, and these clusters are selected as summary candidates. With a user specified summary length, our system generates the corresponding summary by selecting representative frames from candidate clusters and assembling them by their original temporal order. The experimental evaluation demonstrates the effectiveness of our summarization method.
---
paper_title: Automatic generation of video summaries for historical films

paper_content:

A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. In the ECHO (European Chronicles On-line) project, a system was developed to store and manage large collections of historical films for the preservation of cultural heritage. At the University of Mannheim, we have developed the video summarization component of the ECHO system. We discuss the particular challenges the historical film material poses, and how we have designed new video processing algorithms and modified existing ones to cope with noisy black-and-white films.
---
paper_title: VSUM: summarizing from videos

paper_content:

Summarization on produced type of video data (like news or movies) is to find important segments that contain rich information. Users could obtain the important messages by reading summaries rather than full documents. The research in this area could be divided into two parts: (I) image processing (IP) perspective, and (2) NLP (nature language processing) perspective. The former put emphasis on the detection of key frames, while the later focused on the extraction of important concepts. This paper proposes a video summarization system, VSUM. VSUM first identifies all caption words, and then adopts a technique to find the important segments. An external thesaurus is also used in VSUM to enhance the summary extraction process. The experimental results show that VSUM could perform well even if the accuracy of OCR (optical character recognition) is not sophisticating.
---
paper_title: Video summarization and scene detection by graph modeling

paper_content:

We propose a unified approach for video summarization based on the analysis of video structures and video highlights. Two major components in our approach are scene modeling and highlight detection. Scene modeling is achieved by normalized cut algorithm and temporal graph analysis, while highlight detection is accomplished by motion attention modeling. In our proposed approach, a video is represented as a complete undirected graph and the normalized cut algorithm is carried out to globally and optimally partition the graph into video clusters. The resulting clusters form a directed temporal graph and a shortest path algorithm is proposed to efficiently detect video scenes. The attention values are then computed and attached to the scenes, clusters, shots, and subshots in a temporal graph. As a result, the temporal graph can inherently describe the evolution and perceptual importance of a video. In our application, video summaries that emphasize both content balance and perceptual quality can be generated directly from a temporal graph that embeds both the structure and attention information.
---
paper_title: Low cost soccer video summaries based on visual rhythm

paper_content:

The visual rhythm is a spatio-temporal sampled representation of video data providing compact information while preserving several types of video events.We exploit these properties in the present work to propose two new low level descriptors for the analysis of soccer videos computed directly from the visual rhythm.The descriptors are related to dominant color and camera motion estimation.The new descriptors are applied in different tasks aiming the analysis of soccer videos such as shot transition detection, shot classification and attack direction estimation.We also present a simple automated soccer summary application to illustrate the use of the new descriptors.
---
paper_title: Learning personalized video highlights from detailed MPEG-7 metadata

paper_content:

We present a new framework for generating personalized video digests from detailed event metadata. In the new approach high level semantic features (e.g., number of offensive events) are extracted from an existing metadata signal using time windows (e.g., features within 16 sec. intervals). Personalized video digests are generated using a supervised learning algorithm which takes as input examples of important/unimportant events. Window-based features are extracted from the metadata and used to train the system and build a classifier that, given metadata for a new video, classifies segments into important and unimportant, according to a specific user, to generate personalized video digests. Our experimental results using soccer video suggest that extracting high level semantic information from existing metadata can be used effectively (80% precision and 85% recall using cross validation) in generating personalized video digests.
---
paper_title: Video summarization based on the psychological unfolding of a drama

paper_content:

This paper proposes a method of composing digests of TV videos based on their psychological unfolding. Most of the studies of video summarization to date have been based on the video structure, such as cuts or objects in the video. It is generally considered, however, that the observer of a TV drama emphasizes instead the psychological unfolding of the video content (such as the climax). However, no method of summarizing a drama based on the psychological aspect of the video has been proposed. Track structures [such as cutting, lines spoken by the actors, BGM (background music), and sound effects] constitute physical components, and the dramatist skillfully manipulates the time structure based on his empirical knowledge in order to present a particular psychological impression. This paper proposes a method in which the psychologically important part is detected on the basis of empirical knowledge as a corresponding time-series pattern of the track structure, and a summarized video is generated. As an experiment, the proposed method was applied to a TV drama, and the time length was reduced to approximately 1/14. Subjective evaluation experiments show that the proposed method preserves the content of the main plot, with naturalness of extraction, thus indicating the usefulness of the proposed method. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(11): 39–49, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10066
---
paper_title: The holy grail of content-based media analysis

paper_content:

Tools and systems for content-based access to multimedia and-image., video, audio, graphics, text, and any number of combinations-has increased in the last decade. We've seen a common theme of developing automatic analysis techniques for deriving metadata (data describing information in the content at both syntactic and semantic levels). Such metadata facilitates developing innovative tools and systems for multimedia information retrieval, summarization, delivery, and manipulation. Many interesting demonstrations of potential applications and services have emerged-finding images visually similar to a chosen picture (or sketch); summarizing videos with thumbnails of keyframes; finding video clips of a specific event, story, or person; and producing a two-minute skim of an hour-long program. In order to evaluate content-based research methodologies, the article considers intended users and whether alternative solutions exist and areas of research.
---
paper_title: MSN: statistical understanding of broadcasted baseball video using multi-level semantic network

paper_content:

The information processing of sports video yields valuable semantics for content delivery over narrowband networks. Traditional image/video processing is formulated in terms of low-level features describing image/video structure and intensity, while the high-level knowledge such as common sense and human perceptual knowledge are encoded in abstract and nongeometric representations. The management of semantic information in video becomes more and more difficult because of the large difference in representations, levels of knowledge, and abstract episodes. This paper proposes a semantic highlight detection scheme using a Multi-level Semantic Network (MSN) for baseball video interpretation. The probabilistic structure can be applied for highlight detection and shot classification. Satisfactory results will be shown to illustrate better performance compared with the traditional ones.
---
paper_title: Highlights for more complete sports video summarization

paper_content:

Summarization is an essential requirement for achieving a more compact and interesting representation of sports video contents. We propose a framework that integrates highlights into play segments and reveal why we should still retain breaks. Experimental results show that fast detections of whistle sounds, crowd excitement, and text boxes can complement existing techniques for play-breaks and highlights localization.
---
paper_title: MINMAX optimal video summarization

paper_content:

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth and/or power are limited. In this paper, our work is based on a MINMAX optimization formulation with viewing time, frame skip and bit rate constraints. New metrics for missing frame and video summary distortions are introduced. Optimal algorithm based on dynamic programming is presented along with experimental results.
---
paper_title: Interface Design for MyInfo: a Personal News Demonstrator Combining Web and TV Content

paper_content:

This paper describes MyInfo, a novel interface for a personal news demonstrator that processes and combines content from TV and the web. We detail our design process from concept generation to focus group exploration to final design. Our design focuses on three issues: (i) ease-of-use, (ii) video summarization, and (iii) news personalization. With a single button press on the remote control, users access specific topics such as weather or traffic. In addition, users can play back personalized news content as a TV show, leaving themselves free to complete other tasks in their homes, while consuming the news.
---
paper_title: Content-based multimedia information retrieval: State of the art and challenges

paper_content:

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100p recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.
---
paper_title: From context to content: leveraging context to infer media metadata

paper_content:

The recent popularity of mobile camera phones allows for new opportunities to gather important metadata at the point of capture. This paper describes a method for generating metadata for photos using spatial, temporal, and social context. We describe a system we implemented for inferring location information for pictures taken with camera phones and its performance evaluation. We propose that leveraging contextual metadata at the point of capture can address the problems of the semantic and sensory gaps. In particular, combining and sharing spatial, temporal, and social contextual metadata from a given user and across users allows us to make inferences about media content.
---
paper_title: Context and memory in multimedia content analysis

paper_content:

With the advent of broadband networking, video will be available online as well as through traditional distribution channels. The merging of entertainment and information media makes video content classification and retrieval a necessary tool. To provide fast retrieval, content management systems must discern between categories of video. Automatic multimedia analysis techniques for deriving high-level descriptions and annotations have experienced a tremendous surge in interest. Academia and industry have also been challenged to develop realistic applications-from home media library organizers and multimedia lecture archives to broadcast TV content navigators and video-on-demand-in pursuit of the killer application. Current content classification technologies have undoubtedly emerged from traditional image processing and computer vision, audio analysis and processing, and information retrieval. Although terminology varies, the algorithms generally fall into three categories: tangible detectors, high-level abstractors, and latent or intangible descriptors. This paper presents the reflections of the work done by the author and the work ahead.
---
paper_title: Video summarization: methods and landscape

paper_content:

The ability to summarize and abstract information will be an essential part of intelligent behavior in consumer devices. Various summarization methods have been the topic of intensive research in the content-based video analysis community. Summarization in traditional information retrieval is a well understood problem. While there has been a lot of research in the multimedia community there is no agreed upon terminology and classification of the problems in this domain. Although the problem has been researched from different aspects there is usually no distinction between the various dimensions of summarization. The goal of the paper is to provide the basic definitions of widely used terms such as skimming, summarization, and highlighting. The different levels of summarization: local, global, and meta-level are made explicit. We distinguish among the dimensions of task, content, and method and provide an extensive classification model for the same. We map the existing summary extraction approaches in the literature into this model and we classify the aspects of proposed systems in the literature. In addition, we outline the evaluation methods and provide a brief survey. Finally we propose future research directions based on the white spots that we identified by analysis of existing systems in the literature.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Video Summarization for Large Sports Video Archives

paper_content:

Video summarization is defined as creating a shorter video clip or a video poster which includes only the important scenes in the original video streams. In this paper, we propose two methods of generating a summary of arbitrary length for large sports video archives. One is to create a concise video clip by temporally compressing the amount of the video data. The other is to provide a video poster by spatially presenting the image keyframes which together represent the whole video content. Our methods deal with the metadata which has semantic descriptions of video content. Summaries are created according to the significance of each video segment which is normalized in order to handle large sports video archives. We experimentally verified the effectiveness of our methods by comparing the results with man-made video summaries
---
paper_title: Learning personalized video highlights from detailed MPEG-7 metadata

paper_content:

We present a new framework for generating personalized video digests from detailed event metadata. In the new approach high level semantic features (e.g., number of offensive events) are extracted from an existing metadata signal using time windows (e.g., features within 16 sec. intervals). Personalized video digests are generated using a supervised learning algorithm which takes as input examples of important/unimportant events. Window-based features are extracted from the metadata and used to train the system and build a classifier that, given metadata for a new video, classifies segments into important and unimportant, according to a specific user, to generate personalized video digests. Our experimental results using soccer video suggest that extracting high level semantic information from existing metadata can be used effectively (80% precision and 85% recall using cross validation) in generating personalized video digests.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Video Summarization for Large Sports Video Archives

paper_content:

Video summarization is defined as creating a shorter video clip or a video poster which includes only the important scenes in the original video streams. In this paper, we propose two methods of generating a summary of arbitrary length for large sports video archives. One is to create a concise video clip by temporally compressing the amount of the video data. The other is to provide a video poster by spatially presenting the image keyframes which together represent the whole video content. Our methods deal with the metadata which has semantic descriptions of video content. Summaries are created according to the significance of each video segment which is normalized in order to handle large sports video archives. We experimentally verified the effectiveness of our methods by comparing the results with man-made video summaries
---
paper_title: Learning personalized video highlights from detailed MPEG-7 metadata

paper_content:

We present a new framework for generating personalized video digests from detailed event metadata. In the new approach high level semantic features (e.g., number of offensive events) are extracted from an existing metadata signal using time windows (e.g., features within 16 sec. intervals). Personalized video digests are generated using a supervised learning algorithm which takes as input examples of important/unimportant events. Window-based features are extracted from the metadata and used to train the system and build a classifier that, given metadata for a new video, classifies segments into important and unimportant, according to a specific user, to generate personalized video digests. Our experimental results using soccer video suggest that extracting high level semantic information from existing metadata can be used effectively (80% precision and 85% recall using cross validation) in generating personalized video digests.
---
paper_title: Optimizing user expectations for video semantic filtering and abstraction

paper_content:

We describe a novel automatic system that generates personalized videos based on semantic filtering or summarization techniques. This system uses a new set of more than one hundred visual semantic detectors that automatically detect video concepts in faster than realtime. Based on personal profiles, the system generates either video summaries from video databases or filtered video contents from live broadcasting videos. The prototype experiments have shown the effectiveness and stabilities of the system.
---
paper_title: Semantic video summarization in compressed domain MPEG video

paper_content:

In this paper, we present a semantic summarization algorithm that interfaces with the metadata and that works in compressed domain, in particular MPEG-1 and MPEG-2 videos. In enabling a summarization algorithm through high-level semantic content, we try to address two major problems. First, we present the facility provided in the DVA system that allows the semi-automatic creation of this metadata. Second, we address the main point of this system which is the utilization of this metadata to filter out frames, creating an abstract of a video summary quality survey indicates that the proposed method performs satisfactorily.
---
paper_title: Personalized abstraction of broadcasted American football video by highlight selection

paper_content:

Video abstraction is defined as creating shorter video clips or video posters from an original video stream. In this paper, we propose a method of generating a personalized abstract of broadcasted American football video. We first detect significant events in the video stream by matching textual overlays appearing in an image frame with the descriptions of gamestats in which highlights of the game are described. Then, we select highlight shots which should be included in the video abstract from those detected events reflecting on their significance degree and personal preferences, and generate a video clip by connecting the shots augmented with related audio and text. An hour-length video can be compressed into a minute-length personalized abstract. We experimentally verified the effectiveness of this method by comparing man-made video abstracts.
---
paper_title: Automatic video summarizing tool using MPEG-7 descriptors for personal video recorder

paper_content:

We introduce an automatic video summarizing tool (AVST) for a personal video recorder. The tool utilizes MPEG-7 visual descriptors to generate a video index for a summary. The resulting index generates not only a preview of a movie but also allows non-linear access with thumbnails. In addition, the index supports the searching of shots similar to a desired one within saved video sequences. Moreover, simple shot-based video editing can readily be achieved using the generated index.
---
paper_title: Personalized video summary using visual semantic annotations and automatic speech transcriptions

paper_content:

A personalized video summary is dynamically generated in our video personalization and summary system based on user preference and usage environment. The three-tier personalization system adopts the server-middleware-client architecture in order maintain, select, adapt, and deliver rich media content to the user. The server stores the content sources along with their corresponding MPEG-7 metadata descriptions. In this paper, the metadata includes visual semantic annotations and automatic speech transcriptions. Our personalization and summarization engine in the middleware selects the optimal set of desired video segments by matching shot annotations and sentence transcripts with user preferences. The process includes the shot-to-sentence alignment, summary segment selection, and user preference matching and propagation. As a result, the relevant visual shot and audio sentence segments are aggregated and composed into a personalized video summary.
---
paper_title: Media content and type selection from always-on wearable video

paper_content:

A system is described for summarizing head-mounted or hand-carried "always-on" video. The example used is a tourist walking around a historic city with friends and family. The summary consists of a mixture of stills, panoramas and video clips. The system identifies both the scenes to appear in the summary and the media type used to represent them. As there are few shot boundaries in this class of video, the decisions are based on the system's classification of the user's behaviour demonstrated by the motion of the camera, and motion in the scene.
---
paper_title: Video summarization based on user log enhanced link analysis

paper_content:

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
---
paper_title: Tennis video abstraction from audio and visual cues

paper_content:

We propose a context-based model of video abstraction exploiting both audio and video features and applied to tennis TV programs. We can automatically produce different types of summary of a given video depending on the users' constraints or preferences. We have first designed an efficient and accurate temporal segmentation of the video into segments homogeneous w.r.t the camera motion. We introduce original visual descriptors related to the dominant and residual image motions. The different summary types are obtained by specifying adapted classification criteria which involve audio features to select the relevant segments to be included in the video abstract. The proposed scheme has been validated on 22 hours of tennis videos.
---
paper_title: Design and evaluation of a music video summarization system

paper_content:

We present a system that summarizes the textual, audio, and video information of music videos in a format tuned to the preferences of a focus group of 20 users. First, we analyzed user-needs for the content and the layout of the music summaries. Then, we designed algorithms that segment individual song videos from full music video programs by noting changes in color palette, transcript, and audio classification. We summarize each song with automatically selected high level information such as title, artist, duration, title frame, and text as well as audio and visual segments of the chorus. Our system automatically determines with high recall and precision chorus locations, from the placement of repeated words and phrases in the text of the song's lyrics. Our Bayesian belief network then selects other significant video and audio content from the multiple media. Overall, we are able to compress content by a factor of 10. Our second user study has identified the principal variations between users in their choices of content desired in the summary, and in their choices of the platforms that should support their viewing.
---
paper_title: Auto-summarization of audio-video presentations

paper_content:

As streaming audio-video technology becomes widespread, there is a dramatic increase in the amount of multimedia content available on the net. Users face a new challenge: How to examine large amounts of multimedia content quickly. One technique that can enable quick overview of multimedia is video summaries; that is, a shorter version assembled by picking important segments from the original. We evaluate three techniques for automatic creation of summaries for online audio-video presentations. These techniques exploit information in the audio signal (e.g., pitch and pause information), knowledge of slide transition points in the presentation, and information about access patterns of previous users. We report a user study that compares automatically generated summaries that are 20%-25% the length of full presentations to author generated summaries. Users learn from the computer-generated summaries, although less than from authors' summaries. They initially find computer-generated summaries less coherent, but quickly grow accustomed to them.
---
paper_title: Summarizing wearable video

paper_content:

"We want to record our entire life by video" is the motivation of this research. Developing wearable devices and huge storage devices will make it possible to keep entire life by video. We could capture 70 years of our life, however, the problem is how to handle such a huge amount of data. Automatic summarization based on personal interest should be required. In this paper we propose an approach to the automatic structuring and summarization of wearable video. (Wearable video is our abbreviation of "video captured by a wearable camera".) In our approach, we make use of a wearable camera and a sensor of brain waves. The video is firstly structured by objective features of video, and the shots are rated by subjective measures based on brain waves. The approach is very successful for real world experiments and it automatically extracted all the events that the subjects reported they had felt interesting.
---
paper_title: Dynamic video summarization of home video

paper_content:

An increasing number of people own and use camcorders to make videos that capture their experiences and document their lives. These videos easily add up to many hours of material. Oddly, most of them are put into a storage box and never touched or watched again. The reasons for this are manifold. Firstly, the raw video material is unedited, and is therefore long-winded and lacking visually appealing effects. Video editing would help, but, it is still too time-consuming; people rarely find the time to do it. Secondly, watching the same tape more than a few times can be boring, since the video lacks any variation or surprise during playback. Automatic video abstracting algorithms can provide a method for processing videos so that users will want to play the material more often. However, existing automatic abstracting algorithms have been designed for feature films, newscasts or documentaries, and thus are inappropriate for home video material and raw video footage in general. In this paper, we present new algorithms for generating amusing, visually appealing and variable video abstracts of home video material automatically. They make use of a new, empirically motivated approach, also presented in the paper, to cluster time-stamped shots hierarchically into meaningful units. Last but not least, we propose a simple and natural extension of the way people acquire video - so-called on-the-fly annotations - which will allow a completely new set of applications on raw video footage as well as enable better and more selective automatic video abstracts. Moreover, our algorithms are not restricted to home video but can also be applied to raw video footage in general.
---
paper_title: Efficient retrieval of life log based on context and content

paper_content:

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.
---
paper_title: VideoQA: question answering on news video

paper_content:

When querying a news video archive, the users are interested in retrieving precise answers in the form of a summary that best answers the query. However, current video retrieval systems, including the search engines on the web, are designed to retrieve documents instead of precise answers. This research explores the use of question answering (QA) techniques to support personalized news video retrieval. Users interact with our system, VideoQA, using short natural language questions with implicit constraints on contents, context, duration, and genre of expected videos. VideoQA returns short precise news video summaries as answers. The main contributions of this research are: (a) the extension of QA technology to support QA in news video; and (b) the use of multi-modal features, including visual, audio, textual, and external resources, to help correct speech recognition errors and to perform precise question answering. The system has been tested on 7 days of news video and has been found to be effective.
---
paper_title: Using MPEG-7 and MPEG-21 for personalizing video

paper_content:

As multimedia content has proliferated over the past several years, users have begun to expect that content be easily accessed according to their own preferences. One of the most effective ways to do this is through using the MPEG-7 and MPEG-21 standards, which can help address the issues associated with designing a video personalization and summarization system in heterogeneous usage environments. This three-tier architecture provides a standards-compliant infrastructure that, in conjunction with our tools, can help select, adapt, and deliver personalized video summaries to users. In extending our summarization research, we plan to explore semantic similarities across multiple simultaneous news media sources and to abstract summaries for different viewpoints. Doing so will allow us to track a semantic topic as it evolves into the future. As a result, we should be able to summarize news repositories into a smaller collection of topic threads.
---
paper_title: The CPR model for summarizing video

paper_content:

Most past work on video summarization has been based on selecting key frames from videos. We propose a model of video summarization based on three important parameters: Priority (of frames), Continuity (of the summary), and non-Repetition (of the summary). In short, a summary must include high priority frames, must be continuous and non-repetitive. An optimal summary is one that maximizes an objective function based on these three parameters. We develop formal definitions of all these concepts and provide algorithms to find optimal summaries. We briefly report on the performance of these algorithms.
---
paper_title: Framework for personalized multimedia summarization

paper_content:

We extend and validate methods of personalization to the domain of automatically created multimedia summaries. Based on a previously performed user study of 59 people we derived a mapping of personality profile information to preferred multimedia features. This article describes our summarization algorithm. We define constraints for automatic summary generation. Summaries should consist of contiguous segments of full shots, with duration proportional to the log of video length, selected by an objective function of total "importance" of features, with heuristic rules for deciding the "best" combination of length and importance. We validated the summaries with a user study of 32 people. They were asked to answer a shortened series of personality queries. Using this current user profile, together with the earlier genre-specific reduced mapping and with automatically derived features, we automatically generated two summaries for each video: one optimally matched, and one matched to the "opposite" personality. Each user evaluated both summaries on a preference scale for four each of: news, talk show, and music videos. From a statistical analysis we find statistically significant evidence of the effectiveness of personalization on news and music videos, with no evidence of user subpopulations. We conclude for these genres that our claim, of a universal mapping from certain measured personality traits to the computable creation of preferred multimedia summaries, is supported.
---
paper_title: Efficient access to video content in a unified framework

paper_content:

Book contents' browsing and retrieval have been greatly facilitated by its Table-of-Contents (ToC) and Index, respectively. Unfortunately, today's video lacks such powerful mechanisms. In this paper, we explore and present novel techniques for constructing video ToC and Index. Furthermore, we explore the relationship between video browsing and retrieval and propose a unified framework: to incorporate both entities in a seamless way. Experimental results on real-world video clips justify our proposed framework for providing efficient access to video content.
---
paper_title: The priority curve algorithm for video summarization

paper_content:

In this paper, we introduce the concept of a priority curve associated with a video. We then provide an algorithm that can use the priority curve to create a summary (of a desired length) of any video. The summary thus created exhibits nice continuity properties and also avoids repetition. We have implemented the priority curve algorithm (PCA) and compared it with other summarization algorithms in the literature. We show that PCA is faster than existing algorithms and also produces better quality summaries. The quality of summaries was evaluated by a group of 200 students in Naples, Italy, who watched soccer videos. We also briefly describe a soccer video summarization system we have built on using the PCA architecture and various (classical) image processing algorithms.
---
paper_title: Video summarization based on the psychological unfolding of a drama

paper_content:

This paper proposes a method of composing digests of TV videos based on their psychological unfolding. Most of the studies of video summarization to date have been based on the video structure, such as cuts or objects in the video. It is generally considered, however, that the observer of a TV drama emphasizes instead the psychological unfolding of the video content (such as the climax). However, no method of summarizing a drama based on the psychological aspect of the video has been proposed. Track structures [such as cutting, lines spoken by the actors, BGM (background music), and sound effects] constitute physical components, and the dramatist skillfully manipulates the time structure based on his empirical knowledge in order to present a particular psychological impression. This paper proposes a method in which the psychologically important part is detected on the basis of empirical knowledge as a corresponding time-series pattern of the track structure, and a summarized video is generated. As an experiment, the proposed method was applied to a TV drama, and the time length was reduced to approximately 1/14. Subjective evaluation experiments show that the proposed method preserves the content of the main plot, with naturalness of extraction, thus indicating the usefulness of the proposed method. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(11): 39–49, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10066
---
paper_title: Highlights for more complete sports video summarization

paper_content:

Summarization is an essential requirement for achieving a more compact and interesting representation of sports video contents. We propose a framework that integrates highlights into play segments and reveal why we should still retain breaks. Experimental results show that fast detections of whistle sounds, crowd excitement, and text boxes can complement existing techniques for play-breaks and highlights localization.
---
paper_title: Interface Design for MyInfo: a Personal News Demonstrator Combining Web and TV Content

paper_content:

This paper describes MyInfo, a novel interface for a personal news demonstrator that processes and combines content from TV and the web. We detail our design process from concept generation to focus group exploration to final design. Our design focuses on three issues: (i) ease-of-use, (ii) video summarization, and (iii) news personalization. With a single button press on the remote control, users access specific topics such as weather or traffic. In addition, users can play back personalized news content as a TV show, leaving themselves free to complete other tasks in their homes, while consuming the news.
---
paper_title: Optimizing user expectations for video semantic filtering and abstraction

paper_content:

We describe a novel automatic system that generates personalized videos based on semantic filtering or summarization techniques. This system uses a new set of more than one hundred visual semantic detectors that automatically detect video concepts in faster than realtime. Based on personal profiles, the system generates either video summaries from video databases or filtered video contents from live broadcasting videos. The prototype experiments have shown the effectiveness and stabilities of the system.
---
paper_title: Semantic video summarization in compressed domain MPEG video

paper_content:

In this paper, we present a semantic summarization algorithm that interfaces with the metadata and that works in compressed domain, in particular MPEG-1 and MPEG-2 videos. In enabling a summarization algorithm through high-level semantic content, we try to address two major problems. First, we present the facility provided in the DVA system that allows the semi-automatic creation of this metadata. Second, we address the main point of this system which is the utilization of this metadata to filter out frames, creating an abstract of a video summary quality survey indicates that the proposed method performs satisfactorily.
---
paper_title: Personalized abstraction of broadcasted American football video by highlight selection

paper_content:

Video abstraction is defined as creating shorter video clips or video posters from an original video stream. In this paper, we propose a method of generating a personalized abstract of broadcasted American football video. We first detect significant events in the video stream by matching textual overlays appearing in an image frame with the descriptions of gamestats in which highlights of the game are described. Then, we select highlight shots which should be included in the video abstract from those detected events reflecting on their significance degree and personal preferences, and generate a video clip by connecting the shots augmented with related audio and text. An hour-length video can be compressed into a minute-length personalized abstract. We experimentally verified the effectiveness of this method by comparing man-made video abstracts.
---
paper_title: Automatic video summarizing tool using MPEG-7 descriptors for personal video recorder

paper_content:

We introduce an automatic video summarizing tool (AVST) for a personal video recorder. The tool utilizes MPEG-7 visual descriptors to generate a video index for a summary. The resulting index generates not only a preview of a movie but also allows non-linear access with thumbnails. In addition, the index supports the searching of shots similar to a desired one within saved video sequences. Moreover, simple shot-based video editing can readily be achieved using the generated index.
---
paper_title: Personalized video summary using visual semantic annotations and automatic speech transcriptions

paper_content:

A personalized video summary is dynamically generated in our video personalization and summary system based on user preference and usage environment. The three-tier personalization system adopts the server-middleware-client architecture in order maintain, select, adapt, and deliver rich media content to the user. The server stores the content sources along with their corresponding MPEG-7 metadata descriptions. In this paper, the metadata includes visual semantic annotations and automatic speech transcriptions. Our personalization and summarization engine in the middleware selects the optimal set of desired video segments by matching shot annotations and sentence transcripts with user preferences. The process includes the shot-to-sentence alignment, summary segment selection, and user preference matching and propagation. As a result, the relevant visual shot and audio sentence segments are aggregated and composed into a personalized video summary.
---
paper_title: Media content and type selection from always-on wearable video

paper_content:

A system is described for summarizing head-mounted or hand-carried "always-on" video. The example used is a tourist walking around a historic city with friends and family. The summary consists of a mixture of stills, panoramas and video clips. The system identifies both the scenes to appear in the summary and the media type used to represent them. As there are few shot boundaries in this class of video, the decisions are based on the system's classification of the user's behaviour demonstrated by the motion of the camera, and motion in the scene.
---
paper_title: Video summarization based on user log enhanced link analysis

paper_content:

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
---
paper_title: Tennis video abstraction from audio and visual cues

paper_content:

We propose a context-based model of video abstraction exploiting both audio and video features and applied to tennis TV programs. We can automatically produce different types of summary of a given video depending on the users' constraints or preferences. We have first designed an efficient and accurate temporal segmentation of the video into segments homogeneous w.r.t the camera motion. We introduce original visual descriptors related to the dominant and residual image motions. The different summary types are obtained by specifying adapted classification criteria which involve audio features to select the relevant segments to be included in the video abstract. The proposed scheme has been validated on 22 hours of tennis videos.
---
paper_title: Design and evaluation of a music video summarization system

paper_content:

We present a system that summarizes the textual, audio, and video information of music videos in a format tuned to the preferences of a focus group of 20 users. First, we analyzed user-needs for the content and the layout of the music summaries. Then, we designed algorithms that segment individual song videos from full music video programs by noting changes in color palette, transcript, and audio classification. We summarize each song with automatically selected high level information such as title, artist, duration, title frame, and text as well as audio and visual segments of the chorus. Our system automatically determines with high recall and precision chorus locations, from the placement of repeated words and phrases in the text of the song's lyrics. Our Bayesian belief network then selects other significant video and audio content from the multiple media. Overall, we are able to compress content by a factor of 10. Our second user study has identified the principal variations between users in their choices of content desired in the summary, and in their choices of the platforms that should support their viewing.
---
paper_title: Auto-summarization of audio-video presentations

paper_content:

As streaming audio-video technology becomes widespread, there is a dramatic increase in the amount of multimedia content available on the net. Users face a new challenge: How to examine large amounts of multimedia content quickly. One technique that can enable quick overview of multimedia is video summaries; that is, a shorter version assembled by picking important segments from the original. We evaluate three techniques for automatic creation of summaries for online audio-video presentations. These techniques exploit information in the audio signal (e.g., pitch and pause information), knowledge of slide transition points in the presentation, and information about access patterns of previous users. We report a user study that compares automatically generated summaries that are 20%-25% the length of full presentations to author generated summaries. Users learn from the computer-generated summaries, although less than from authors' summaries. They initially find computer-generated summaries less coherent, but quickly grow accustomed to them.
---
paper_title: Summarizing wearable video

paper_content:

"We want to record our entire life by video" is the motivation of this research. Developing wearable devices and huge storage devices will make it possible to keep entire life by video. We could capture 70 years of our life, however, the problem is how to handle such a huge amount of data. Automatic summarization based on personal interest should be required. In this paper we propose an approach to the automatic structuring and summarization of wearable video. (Wearable video is our abbreviation of "video captured by a wearable camera".) In our approach, we make use of a wearable camera and a sensor of brain waves. The video is firstly structured by objective features of video, and the shots are rated by subjective measures based on brain waves. The approach is very successful for real world experiments and it automatically extracted all the events that the subjects reported they had felt interesting.
---
paper_title: Dynamic video summarization of home video

paper_content:

An increasing number of people own and use camcorders to make videos that capture their experiences and document their lives. These videos easily add up to many hours of material. Oddly, most of them are put into a storage box and never touched or watched again. The reasons for this are manifold. Firstly, the raw video material is unedited, and is therefore long-winded and lacking visually appealing effects. Video editing would help, but, it is still too time-consuming; people rarely find the time to do it. Secondly, watching the same tape more than a few times can be boring, since the video lacks any variation or surprise during playback. Automatic video abstracting algorithms can provide a method for processing videos so that users will want to play the material more often. However, existing automatic abstracting algorithms have been designed for feature films, newscasts or documentaries, and thus are inappropriate for home video material and raw video footage in general. In this paper, we present new algorithms for generating amusing, visually appealing and variable video abstracts of home video material automatically. They make use of a new, empirically motivated approach, also presented in the paper, to cluster time-stamped shots hierarchically into meaningful units. Last but not least, we propose a simple and natural extension of the way people acquire video - so-called on-the-fly annotations - which will allow a completely new set of applications on raw video footage as well as enable better and more selective automatic video abstracts. Moreover, our algorithms are not restricted to home video but can also be applied to raw video footage in general.
---
paper_title: Efficient retrieval of life log based on context and content

paper_content:

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.
---
paper_title: VideoQA: question answering on news video

paper_content:

When querying a news video archive, the users are interested in retrieving precise answers in the form of a summary that best answers the query. However, current video retrieval systems, including the search engines on the web, are designed to retrieve documents instead of precise answers. This research explores the use of question answering (QA) techniques to support personalized news video retrieval. Users interact with our system, VideoQA, using short natural language questions with implicit constraints on contents, context, duration, and genre of expected videos. VideoQA returns short precise news video summaries as answers. The main contributions of this research are: (a) the extension of QA technology to support QA in news video; and (b) the use of multi-modal features, including visual, audio, textual, and external resources, to help correct speech recognition errors and to perform precise question answering. The system has been tested on 7 days of news video and has been found to be effective.
---
paper_title: Using MPEG-7 and MPEG-21 for personalizing video

paper_content:

As multimedia content has proliferated over the past several years, users have begun to expect that content be easily accessed according to their own preferences. One of the most effective ways to do this is through using the MPEG-7 and MPEG-21 standards, which can help address the issues associated with designing a video personalization and summarization system in heterogeneous usage environments. This three-tier architecture provides a standards-compliant infrastructure that, in conjunction with our tools, can help select, adapt, and deliver personalized video summaries to users. In extending our summarization research, we plan to explore semantic similarities across multiple simultaneous news media sources and to abstract summaries for different viewpoints. Doing so will allow us to track a semantic topic as it evolves into the future. As a result, we should be able to summarize news repositories into a smaller collection of topic threads.
---
paper_title: The CPR model for summarizing video

paper_content:

Most past work on video summarization has been based on selecting key frames from videos. We propose a model of video summarization based on three important parameters: Priority (of frames), Continuity (of the summary), and non-Repetition (of the summary). In short, a summary must include high priority frames, must be continuous and non-repetitive. An optimal summary is one that maximizes an objective function based on these three parameters. We develop formal definitions of all these concepts and provide algorithms to find optimal summaries. We briefly report on the performance of these algorithms.
---
paper_title: Framework for personalized multimedia summarization

paper_content:

We extend and validate methods of personalization to the domain of automatically created multimedia summaries. Based on a previously performed user study of 59 people we derived a mapping of personality profile information to preferred multimedia features. This article describes our summarization algorithm. We define constraints for automatic summary generation. Summaries should consist of contiguous segments of full shots, with duration proportional to the log of video length, selected by an objective function of total "importance" of features, with heuristic rules for deciding the "best" combination of length and importance. We validated the summaries with a user study of 32 people. They were asked to answer a shortened series of personality queries. Using this current user profile, together with the earlier genre-specific reduced mapping and with automatically derived features, we automatically generated two summaries for each video: one optimally matched, and one matched to the "opposite" personality. Each user evaluated both summaries on a preference scale for four each of: news, talk show, and music videos. From a statistical analysis we find statistically significant evidence of the effectiveness of personalization on news and music videos, with no evidence of user subpopulations. We conclude for these genres that our claim, of a universal mapping from certain measured personality traits to the computable creation of preferred multimedia summaries, is supported.
---
paper_title: Efficient access to video content in a unified framework

paper_content:

Book contents' browsing and retrieval have been greatly facilitated by its Table-of-Contents (ToC) and Index, respectively. Unfortunately, today's video lacks such powerful mechanisms. In this paper, we explore and present novel techniques for constructing video ToC and Index. Furthermore, we explore the relationship between video browsing and retrieval and propose a unified framework: to incorporate both entities in a seamless way. Experimental results on real-world video clips justify our proposed framework for providing efficient access to video content.
---
paper_title: The priority curve algorithm for video summarization

paper_content:

In this paper, we introduce the concept of a priority curve associated with a video. We then provide an algorithm that can use the priority curve to create a summary (of a desired length) of any video. The summary thus created exhibits nice continuity properties and also avoids repetition. We have implemented the priority curve algorithm (PCA) and compared it with other summarization algorithms in the literature. We show that PCA is faster than existing algorithms and also produces better quality summaries. The quality of summaries was evaluated by a group of 200 students in Naples, Italy, who watched soccer videos. We also briefly describe a soccer video summarization system we have built on using the PCA architecture and various (classical) image processing algorithms.
---
paper_title: Video summarization based on the psychological unfolding of a drama

paper_content:

This paper proposes a method of composing digests of TV videos based on their psychological unfolding. Most of the studies of video summarization to date have been based on the video structure, such as cuts or objects in the video. It is generally considered, however, that the observer of a TV drama emphasizes instead the psychological unfolding of the video content (such as the climax). However, no method of summarizing a drama based on the psychological aspect of the video has been proposed. Track structures [such as cutting, lines spoken by the actors, BGM (background music), and sound effects] constitute physical components, and the dramatist skillfully manipulates the time structure based on his empirical knowledge in order to present a particular psychological impression. This paper proposes a method in which the psychologically important part is detected on the basis of empirical knowledge as a corresponding time-series pattern of the track structure, and a summarized video is generated. As an experiment, the proposed method was applied to a TV drama, and the time length was reduced to approximately 1/14. Subjective evaluation experiments show that the proposed method preserves the content of the main plot, with naturalness of extraction, thus indicating the usefulness of the proposed method. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(11): 39–49, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10066
---
paper_title: Highlights for more complete sports video summarization

paper_content:

Summarization is an essential requirement for achieving a more compact and interesting representation of sports video contents. We propose a framework that integrates highlights into play segments and reveal why we should still retain breaks. Experimental results show that fast detections of whistle sounds, crowd excitement, and text boxes can complement existing techniques for play-breaks and highlights localization.
---
paper_title: Interface Design for MyInfo: a Personal News Demonstrator Combining Web and TV Content

paper_content:

This paper describes MyInfo, a novel interface for a personal news demonstrator that processes and combines content from TV and the web. We detail our design process from concept generation to focus group exploration to final design. Our design focuses on three issues: (i) ease-of-use, (ii) video summarization, and (iii) news personalization. With a single button press on the remote control, users access specific topics such as weather or traffic. In addition, users can play back personalized news content as a TV show, leaving themselves free to complete other tasks in their homes, while consuming the news.
---
paper_title: Affective content detection using HMMs

paper_content:

This paper discusses a new technique for detecting affective events using Hidden Markov Models(HMM). To map low level features of video data to high level emotional events, we perform empirical study on the relationship between emotional events and low-level features. After that, we compute simple low-level features that represent emotional characteristics and construct a token or observation vector by combining low level features. The observation vector sequence is tested to detect emotional events through HMMs. We create two HMM topologies and test both topologies. The affective events are detected from our proposed models with good accuracy.
---
paper_title: Optimizing user expectations for video semantic filtering and abstraction

paper_content:

We describe a novel automatic system that generates personalized videos based on semantic filtering or summarization techniques. This system uses a new set of more than one hundred visual semantic detectors that automatically detect video concepts in faster than realtime. Based on personal profiles, the system generates either video summaries from video databases or filtered video contents from live broadcasting videos. The prototype experiments have shown the effectiveness and stabilities of the system.
---
paper_title: Semantic video summarization in compressed domain MPEG video

paper_content:

In this paper, we present a semantic summarization algorithm that interfaces with the metadata and that works in compressed domain, in particular MPEG-1 and MPEG-2 videos. In enabling a summarization algorithm through high-level semantic content, we try to address two major problems. First, we present the facility provided in the DVA system that allows the semi-automatic creation of this metadata. Second, we address the main point of this system which is the utilization of this metadata to filter out frames, creating an abstract of a video summary quality survey indicates that the proposed method performs satisfactorily.
---
paper_title: Personalized abstraction of broadcasted American football video by highlight selection

paper_content:

Video abstraction is defined as creating shorter video clips or video posters from an original video stream. In this paper, we propose a method of generating a personalized abstract of broadcasted American football video. We first detect significant events in the video stream by matching textual overlays appearing in an image frame with the descriptions of gamestats in which highlights of the game are described. Then, we select highlight shots which should be included in the video abstract from those detected events reflecting on their significance degree and personal preferences, and generate a video clip by connecting the shots augmented with related audio and text. An hour-length video can be compressed into a minute-length personalized abstract. We experimentally verified the effectiveness of this method by comparing man-made video abstracts.
---
paper_title: Content-based multimedia information retrieval: State of the art and challenges

paper_content:

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100p recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.
---
paper_title: Automatic video summarizing tool using MPEG-7 descriptors for personal video recorder

paper_content:

We introduce an automatic video summarizing tool (AVST) for a personal video recorder. The tool utilizes MPEG-7 visual descriptors to generate a video index for a summary. The resulting index generates not only a preview of a movie but also allows non-linear access with thumbnails. In addition, the index supports the searching of shots similar to a desired one within saved video sequences. Moreover, simple shot-based video editing can readily be achieved using the generated index.
---
paper_title: Personalized video summary using visual semantic annotations and automatic speech transcriptions

paper_content:

A personalized video summary is dynamically generated in our video personalization and summary system based on user preference and usage environment. The three-tier personalization system adopts the server-middleware-client architecture in order maintain, select, adapt, and deliver rich media content to the user. The server stores the content sources along with their corresponding MPEG-7 metadata descriptions. In this paper, the metadata includes visual semantic annotations and automatic speech transcriptions. Our personalization and summarization engine in the middleware selects the optimal set of desired video segments by matching shot annotations and sentence transcripts with user preferences. The process includes the shot-to-sentence alignment, summary segment selection, and user preference matching and propagation. As a result, the relevant visual shot and audio sentence segments are aggregated and composed into a personalized video summary.
---
paper_title: Media content and type selection from always-on wearable video

paper_content:

A system is described for summarizing head-mounted or hand-carried "always-on" video. The example used is a tourist walking around a historic city with friends and family. The summary consists of a mixture of stills, panoramas and video clips. The system identifies both the scenes to appear in the summary and the media type used to represent them. As there are few shot boundaries in this class of video, the decisions are based on the system's classification of the user's behaviour demonstrated by the motion of the camera, and motion in the scene.
---
paper_title: Video summarization based on user log enhanced link analysis

paper_content:

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
---
paper_title: Tennis video abstraction from audio and visual cues

paper_content:

We propose a context-based model of video abstraction exploiting both audio and video features and applied to tennis TV programs. We can automatically produce different types of summary of a given video depending on the users' constraints or preferences. We have first designed an efficient and accurate temporal segmentation of the video into segments homogeneous w.r.t the camera motion. We introduce original visual descriptors related to the dominant and residual image motions. The different summary types are obtained by specifying adapted classification criteria which involve audio features to select the relevant segments to be included in the video abstract. The proposed scheme has been validated on 22 hours of tennis videos.
---
paper_title: Affective video content representation and modeling

paper_content:

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.
---
paper_title: Evaluation of video summarization for a large number of cameras in ubiquitous home

paper_content:

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
---
paper_title: Auto-summarization of audio-video presentations

paper_content:

As streaming audio-video technology becomes widespread, there is a dramatic increase in the amount of multimedia content available on the net. Users face a new challenge: How to examine large amounts of multimedia content quickly. One technique that can enable quick overview of multimedia is video summaries; that is, a shorter version assembled by picking important segments from the original. We evaluate three techniques for automatic creation of summaries for online audio-video presentations. These techniques exploit information in the audio signal (e.g., pitch and pause information), knowledge of slide transition points in the presentation, and information about access patterns of previous users. We report a user study that compares automatically generated summaries that are 20%-25% the length of full presentations to author generated summaries. Users learn from the computer-generated summaries, although less than from authors' summaries. They initially find computer-generated summaries less coherent, but quickly grow accustomed to them.
---
paper_title: Summarizing wearable video

paper_content:

"We want to record our entire life by video" is the motivation of this research. Developing wearable devices and huge storage devices will make it possible to keep entire life by video. We could capture 70 years of our life, however, the problem is how to handle such a huge amount of data. Automatic summarization based on personal interest should be required. In this paper we propose an approach to the automatic structuring and summarization of wearable video. (Wearable video is our abbreviation of "video captured by a wearable camera".) In our approach, we make use of a wearable camera and a sensor of brain waves. The video is firstly structured by objective features of video, and the shots are rated by subjective measures based on brain waves. The approach is very successful for real world experiments and it automatically extracted all the events that the subjects reported they had felt interesting.
---
paper_title: Hierarchical video summarization based on context clustering

paper_content:

A personalized video summary is dynamically generated in our video personalization and summarization system based on user preference and usage environment. The three-tier personalization system adopts the server-middleware-client architecture in order to maintain, select, adapt, and deliver rich media content to the user. The server stores the content sources along with their corresponding MPEG-7 metadata descriptions. In this paper, the metadata includes visual semantic annotations and automatic speech transcriptions. Our personalization and summarization engine in the middleware selects the optimal set of desired video segments by matching shot annotations and sentence transcripts with user preferences. Besides finding the desired contents, the objective is to present a coherent summary. There are diverse methods for creating summaries, and we focus on the challenges of generating a hierarchical video summary based on context information. In our summarization algorithm, three inputs are used to generate the hierarchical video summary output. These inputs are (1) MPEG-7 metadata descriptions of the contents in the server, (2) user preference and usage environment declarations from the user client, and (3) context information including MPEG-7 controlled term list and classification scheme. In a video sequence, descriptions and relevance scores are assigned to each shot. Based on these shot descriptions, context clustering is performed to collect consecutively similar shots to correspond to hierarchical scene representations. The context clustering is based on the available context information, and may be derived from domain knowledge or rules engines. Finally, the selection of structured video segments to generate the hierarchical summary efficiently balances between scene representation and shot selection.
---
paper_title: Dynamic video summarization of home video

paper_content:

An increasing number of people own and use camcorders to make videos that capture their experiences and document their lives. These videos easily add up to many hours of material. Oddly, most of them are put into a storage box and never touched or watched again. The reasons for this are manifold. Firstly, the raw video material is unedited, and is therefore long-winded and lacking visually appealing effects. Video editing would help, but, it is still too time-consuming; people rarely find the time to do it. Secondly, watching the same tape more than a few times can be boring, since the video lacks any variation or surprise during playback. Automatic video abstracting algorithms can provide a method for processing videos so that users will want to play the material more often. However, existing automatic abstracting algorithms have been designed for feature films, newscasts or documentaries, and thus are inappropriate for home video material and raw video footage in general. In this paper, we present new algorithms for generating amusing, visually appealing and variable video abstracts of home video material automatically. They make use of a new, empirically motivated approach, also presented in the paper, to cluster time-stamped shots hierarchically into meaningful units. Last but not least, we propose a simple and natural extension of the way people acquire video - so-called on-the-fly annotations - which will allow a completely new set of applications on raw video footage as well as enable better and more selective automatic video abstracts. Moreover, our algorithms are not restricted to home video but can also be applied to raw video footage in general.
---
paper_title: Efficient retrieval of life log based on context and content

paper_content:

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.
---
paper_title: Using MPEG-7 and MPEG-21 for personalizing video

paper_content:

As multimedia content has proliferated over the past several years, users have begun to expect that content be easily accessed according to their own preferences. One of the most effective ways to do this is through using the MPEG-7 and MPEG-21 standards, which can help address the issues associated with designing a video personalization and summarization system in heterogeneous usage environments. This three-tier architecture provides a standards-compliant infrastructure that, in conjunction with our tools, can help select, adapt, and deliver personalized video summaries to users. In extending our summarization research, we plan to explore semantic similarities across multiple simultaneous news media sources and to abstract summaries for different viewpoints. Doing so will allow us to track a semantic topic as it evolves into the future. As a result, we should be able to summarize news repositories into a smaller collection of topic threads.
---
paper_title: The CPR model for summarizing video

paper_content:

Most past work on video summarization has been based on selecting key frames from videos. We propose a model of video summarization based on three important parameters: Priority (of frames), Continuity (of the summary), and non-Repetition (of the summary). In short, a summary must include high priority frames, must be continuous and non-repetitive. An optimal summary is one that maximizes an objective function based on these three parameters. We develop formal definitions of all these concepts and provide algorithms to find optimal summaries. We briefly report on the performance of these algorithms.
---
paper_title: Framework for personalized multimedia summarization

paper_content:

We extend and validate methods of personalization to the domain of automatically created multimedia summaries. Based on a previously performed user study of 59 people we derived a mapping of personality profile information to preferred multimedia features. This article describes our summarization algorithm. We define constraints for automatic summary generation. Summaries should consist of contiguous segments of full shots, with duration proportional to the log of video length, selected by an objective function of total "importance" of features, with heuristic rules for deciding the "best" combination of length and importance. We validated the summaries with a user study of 32 people. They were asked to answer a shortened series of personality queries. Using this current user profile, together with the earlier genre-specific reduced mapping and with automatically derived features, we automatically generated two summaries for each video: one optimally matched, and one matched to the "opposite" personality. Each user evaluated both summaries on a preference scale for four each of: news, talk show, and music videos. From a statistical analysis we find statistically significant evidence of the effectiveness of personalization on news and music videos, with no evidence of user subpopulations. We conclude for these genres that our claim, of a universal mapping from certain measured personality traits to the computable creation of preferred multimedia summaries, is supported.
---
paper_title: Efficient access to video content in a unified framework

paper_content:

Book contents' browsing and retrieval have been greatly facilitated by its Table-of-Contents (ToC) and Index, respectively. Unfortunately, today's video lacks such powerful mechanisms. In this paper, we explore and present novel techniques for constructing video ToC and Index. Furthermore, we explore the relationship between video browsing and retrieval and propose a unified framework: to incorporate both entities in a seamless way. Experimental results on real-world video clips justify our proposed framework for providing efficient access to video content.
---
paper_title: The priority curve algorithm for video summarization

paper_content:

In this paper, we introduce the concept of a priority curve associated with a video. We then provide an algorithm that can use the priority curve to create a summary (of a desired length) of any video. The summary thus created exhibits nice continuity properties and also avoids repetition. We have implemented the priority curve algorithm (PCA) and compared it with other summarization algorithms in the literature. We show that PCA is faster than existing algorithms and also produces better quality summaries. The quality of summaries was evaluated by a group of 200 students in Naples, Italy, who watched soccer videos. We also briefly describe a soccer video summarization system we have built on using the PCA architecture and various (classical) image processing algorithms.
---
paper_title: Learning personalized video highlights from detailed MPEG-7 metadata

paper_content:

We present a new framework for generating personalized video digests from detailed event metadata. In the new approach high level semantic features (e.g., number of offensive events) are extracted from an existing metadata signal using time windows (e.g., features within 16 sec. intervals). Personalized video digests are generated using a supervised learning algorithm which takes as input examples of important/unimportant events. Window-based features are extracted from the metadata and used to train the system and build a classifier that, given metadata for a new video, classifies segments into important and unimportant, according to a specific user, to generate personalized video digests. Our experimental results using soccer video suggest that extracting high level semantic information from existing metadata can be used effectively (80% precision and 85% recall using cross validation) in generating personalized video digests.
---
paper_title: Video summarization based on the psychological unfolding of a drama

paper_content:

This paper proposes a method of composing digests of TV videos based on their psychological unfolding. Most of the studies of video summarization to date have been based on the video structure, such as cuts or objects in the video. It is generally considered, however, that the observer of a TV drama emphasizes instead the psychological unfolding of the video content (such as the climax). However, no method of summarizing a drama based on the psychological aspect of the video has been proposed. Track structures [such as cutting, lines spoken by the actors, BGM (background music), and sound effects] constitute physical components, and the dramatist skillfully manipulates the time structure based on his empirical knowledge in order to present a particular psychological impression. This paper proposes a method in which the psychologically important part is detected on the basis of empirical knowledge as a corresponding time-series pattern of the track structure, and a summarized video is generated. As an experiment, the proposed method was applied to a TV drama, and the time length was reduced to approximately 1/14. Subjective evaluation experiments show that the proposed method preserves the content of the main plot, with naturalness of extraction, thus indicating the usefulness of the proposed method. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(11): 39–49, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10066
---
paper_title: Highlights for more complete sports video summarization

paper_content:

Summarization is an essential requirement for achieving a more compact and interesting representation of sports video contents. We propose a framework that integrates highlights into play segments and reveal why we should still retain breaks. Experimental results show that fast detections of whistle sounds, crowd excitement, and text boxes can complement existing techniques for play-breaks and highlights localization.
---
paper_title: Interface Design for MyInfo: a Personal News Demonstrator Combining Web and TV Content

paper_content:

This paper describes MyInfo, a novel interface for a personal news demonstrator that processes and combines content from TV and the web. We detail our design process from concept generation to focus group exploration to final design. Our design focuses on three issues: (i) ease-of-use, (ii) video summarization, and (iii) news personalization. With a single button press on the remote control, users access specific topics such as weather or traffic. In addition, users can play back personalized news content as a TV show, leaving themselves free to complete other tasks in their homes, while consuming the news.
---

Title: Video summarisation: A conceptual framework and survey of the state of the art

Section 1: Introduction
Description 1: Introduce the growing need for video summarisation and give an overview of the paper's goals and structure.

Section 2: A conceptual framework for video summarisation
Description 2: Present a conceptual framework for video summarisation, including the rationale and basis for categorising video summarisation techniques and summaries.

Section 3: Techniques for summarising video
Description 3: Describe the different techniques for video summarisation classified into internal, external, and hybrid techniques.

Section 4: Video summaries produced by video summarisation techniques
Description 4: Discuss the various types of video summaries produced by internal, external, and hybrid techniques.

Section 5: Applications of video summarisation
Description 5: Outline different applications of video summarisation and how they leverage the benefits of summarised video content.

Section 6: Internal summarisation techniques and summaries
Description 6: Review internal summarisation techniques and the types of summaries they produce.

Section 7: External summarisation techniques and summaries
Description 7: Describe external summarisation techniques, how they operate, and the kinds of summaries they generate.

Section 8: Hybrid summarisation techniques and summaries
Description 8: Discuss hybrid summarisation techniques and the resultant summaries they produce by combining both internal and external information.

Section 9: Challenges and recommendations for future directions
Description 9: Identify challenges faced by current summarisation techniques and provide recommendations for future research directions in video summarisation.

Section 10: Concluding discussion
Description 10: Summarise the key points discussed in the paper and the importance of future research directions to address existing challenges in video summarisation.