A survey on automatic techniques for enhancement and analysis of digital photography
---
paper_title: Image Quality Assessment Based on Structural Distortion and Image Definition

paper_content:

Image quality assessment is one of the key technologies in image processing field. The factors of the image quality assessment include gradation, definition and consistency. The image quality assessment method based on structural distortion often conclude distortion problem into three different factors: loss of correlation, mean distortion, and variance distortion and the image definition factor is not taken into account. In this paper, an image quality assessment method based on structural distortion and image definition is developed. The novel index of image quality assessment based on structural distortion and image definition is also proposed. The experimental results prove the image quality assessment method will be more consistent with human perception.
---
paper_title: Image quality assessment based on wave atoms transform

paper_content:

Image quality assessment is still an active field of research. The main objective of the developed image quality metric is to offer an index of quality that is consistent with the human subjective judgment of image quality. Despite the great number of developed metrics, there is still a need for image analysis tools that is able to extract the most perceptual relevant characteristics of an image. The goal of this work is then to propose a more advanced analysis and representation tools to extract more effective features that could be incorporated in the design of the image quality metric. In this paper, we propose a novel objective metric based on wave atoms transform. This new transform is half multiscale and half multi-directional. It offers a better representation of images containing oscillatory patterns and textures than the others known transforms [10]. In this work, we propose a new full reference image quality metric based on wave atom transform and exploiting some properties of the human visual system. The consistency of the proposed metric with subjective evaluation is performed on LIVE database. The obtained correlation of this metric with the MOS provided by the database is better than other known metrics confirming thus the efficiency of this new image quality measure in predicting image quality.
---
paper_title: A framework for photo-quality assessment and enhancement based on visual aesthetics

paper_content:

We present an interactive application that enables users to improve the visual aesthetics of their digital photographs using spatial recomposition. Unlike earlier work that focuses either on photo quality assessment or interactive tools for photo editing, we enable the user to make informed decisions about improving the composition of a photograph and to implement them in a single framework. Specifically, the user interactively selects a foreground object and the system presents recommendations for where it can be moved in a manner that optimizes a learned aesthetic metric while obeying semantic constraints. For photographic compositions that lack a distinct foreground object, our tool provides the user with cropping or expanding recommendations that improve its aesthetic quality. We learn a support vector regression model for capturing image aesthetics from user data and seek to optimize this metric during recomposition. Rather than prescribing a fully-automated solution, we allow user-guided object segmentation and inpainting to ensure that the final photograph matches the user's criteria. Our approach achieves 86% accuracy in predicting the attractiveness of unrated images, when compared to their respective human rankings. Additionally, 73% of the images recomposited using our tool are ranked more attractive than their original counterparts by human raters.
---
paper_title: Content-aware photo-on-photo composition for consumer photos

paper_content:

We propose an approach to automatically generate photo-on-photo compositions using only a consumer's own image collection, without imposing a strong theme constraint dictated by typical background art designs. Our goal is to ensure adequate foreground and background photo sizing, avoid occlusion of important background photo content, and respect simple aesthetic rules in photo arrangement. We propose a novel image content redundancy measure to extract the informative zone, which is a rectangular region of the background photo that should not be occluded by foreground content. Background photo layout and foreground layout regions are then calculated based on a placement algorithm that deals with possible aspect ratio mismatch of the canvas and background photo, as well as cases of insufficient foreground photo layout area.
---
paper_title: Evaluating visual aesthetics in photographic portraiture

paper_content:

We propose and demonstrate a strategy to quantify aesthetic quality in photographs. Our approach is to develop a small set of classification features by tuning general compositional principles to a targeted image domain where saliency can be better understood. We demonstrate this strategy with photographic portraits of individuals, but it can be extended to other domains. Our technique leverages a refined method of using templates as spatial composition feature look-up tables. Compared to the traditional approach using a large set of global and local features extracted with little salient knowledge, classifiers using features extracted with our approach are better predictors of human aesthetic judgments.
---
paper_title: MediAssist: using content-based analysis and context to manage personal photo collections

paper_content:

We present work which organises personal digital photo collections based on contextual information, such as time and location, combined with content-based analysis such as face detection and other feature detectors. The MediAssist demonstration system illustrates the results of our research into digital photo management, showing how a combination of automatically extracted context and content-based information, together with user annotation, facilitates efficient searching of personal photo collections.
---
paper_title: Scenic photo quality assessment with bag of aesthetics-preserving features

paper_content:

In this paper, an aesthetic modeling method for scenic photographs is proposed. A bottom-up approach is developed to construct an aesthetic library with bag-of-aesthetics preserving features instead of top-down methods that implement the heuristic guidelines (rule-specific features) listed in the photography literature, which is employed in previous works. The proposed method can cover both implicit and explicit aesthetic features with a learning process. The experimental results show that the proposed features in the library (92.06% in accuracy) outperform the state-of-the-art rule-specific features (83.63% in accuracy) significantly in the aesthetic quality assessment for scenic photos, and the rule-specific features are also proved to be encompassed by the proposed features. Meanwhile, it is observed from experiments that the features extracted for contrast information are more effective than those for absolute information, which is consistent with the properties of human visual systems.
---
paper_title: PhotoCube: effective and efficient multi-dimensional browsing of personal photo collections

paper_content:

It has never been so easy to take pictures, and personal image collections have never been so large. Unfortunately, most current photo browsers provide very limited support for effectively navigating image collections. This demonstration proposal describes PhotoCube, a personal image browser based on a multi-dimensional data model similar to the model used in OLAP applications. With PhotoCube, users can tag pictures, structure tags into various hierarchies, and browse images according to any possible perspective. We also describe three demonstration scenarios that show the power, flexibility and scalability of PhotoCube.
---
paper_title: Summarization of Archived and Shared Personal Photo Collections

paper_content:

The volume of personal photos hosted on photo archives and social sharing platforms has been increasing exponentially. It is difficult to get an overview of a large collection of personal photos without browsing though the entire database manually. In this research, we propose a framework to generate representative subset summaries from photo collections hosted on web archives or social networks. We define salient properties of an effective photo summary and model summarization as an optimization of these properties, given the size constraints. We also introduce metrics for evaluating photo summaries based on their information content and the ability to satisfy user's information needs. Our experiments show that our summarization framework performs better than baseline algorithms.
---
paper_title: Subjective image quality assessment based on objective image quality measurement factors

paper_content:

This study was based on the prevailing need to reproduce customers' preferred images. Therefore, we propose a new subjective image quality assessment consisting of languages that customers and manufacturers can easily understand based on the objective image quality measurement factors. In order to examine subjective image quality assessment, these objective image quality measurement factors are expressed by general adjective language throughout the experimental process. To draw adjective languages, the objective image quality measurement factors are presented by stimuli, and subjects expressed their perception of an image quality. The resulting languages are used to question an image quality. According to the analysis, there are meaningful correlations between image quality measurement factors. Moreover, the results indicated that items which strongly affect preference are color reproduction, dynamic range, resolution, and noise, respectively. The study concurs with Engeldrum's The Complete Image Circle and affirms the positive role of interaction between manufacturers and consumers to assess and enhance image quality preference and equipment performance.
---
paper_title: Comparison of image quality assessment algorithms on compressed images

paper_content:

A crucial step in image compression is the evaluation of its performance, and more precisely the available way ::: to measure the final quality of the compressed image. Usually, to measure performance, some measure of the ::: covariation between the subjective ratings and the degree of compression is performed between rated image ::: quality and algorithm. Nevertheless, local variations are not well taken into account. ::: We use the recently introduced Maximum Likelihood Difference Scaling (MLDS) method to quantify suprathreshold ::: perceptual differences between pairs of images and examine how perceived image quality estimated ::: through MLDS changes the compression rate is increased. This approach circumvents the limitations inherent ::: to subjective rating methods.
---
paper_title: Creating animation with personal photo collections and map for storytelling

paper_content:

The goal of our research is to support the sharing of stories with digital photographs. Some map sites are now collecting stories associated with peoples' relationships to places. Users are mapping collections of places including intangible emotional associations of places along with photographs, videos. Though this framework of mapping stories is important for accelerating individual creation and transmission of map content, it is not expressive enough to communicate stories narratively. For example, especially when the number of the mapped collections of places is large, it is not easy for viewers to read the map, and it is not easy for creators to express stories as series of events in the real world. That is because one narrative story in a form of a sequence of text narrations, a sequence of photographs, a movie, and an audio like podcasting, etc. is mapped to just one point. As a result, it's up to the viewers which point on the map to read, and in what order. The common framework is rather suitable for mapping and expressing fragments or snapshots of a whole story, and not suitable for expressing a whole story narratively by using a whole area of a map as the setting of the story. We therefore propose a new framework for mapping personal photo collections and constructing them as stories such as route guidances, sightseeing guidances, historical topics, fieldwork records, personal diaries, and so on. We named this framework as spatial slideshow. A spatial slideshow is a fusion of personal photo mapping and photo storytelling. Each story of a spatial slideshow is made of a sequence of mapped photographs, and presented as synchronized animations of a map and an enhanced photo slideshow. The main technical suggestion of this paper is a method for creating a three-dimensional animation of photo image which has a visual effect of moving from photo to photo. We have developed a personal photo album software and released it on the Web, which works as an editor and a browser of spatial slideshows. We assume the proposed framework has a significance of helping with grassroots development of spatial content driven by visual communications about places or events in the real world.
---
paper_title: Structural information-based image quality assessment using LU factorization

paper_content:

The goal of the objective image quality assessment is to quantitatively measure the image quality of an arbitrary image. The objective image quality measure is desirable if it is close to the subjective image quality assessment such as the mean opinion score. Image quality assessment algorithms are generally classified into two methodologies: perceptual and structural information-based. This paper proposes a structural information-based image quality assessment algorithm, in which LU factorization is used for representation of the structural information of an image. The proposed algorithm performs LU factorization of each of reference and distorted images, from which the distortion map is computed for measuring the quality of the distorted image. Finally, the proposed image quality metric is computed from the two-dimensional distortion map. Experimental results with the laboratory for image and video engineering database images show the efficiency of the proposed method, calibrated by linear and logistic regressions, in terms of the Pearson correlation coefficient and root mean square error. In commercial systems, the proposed algorithm can be used for quality assessment of mobile contents and video coding, which effectively replaces the peak signal to noise ratio or the mean square error.
---
paper_title: A tangible interface for browsing digital photo collections

paper_content:

We present a design concept of a tangible user interface for browsing image contents. A layer structure for image presentation and three kinematical gestures are proposed to facilitate navigation in the digital photo collections. We describe how gestures support the photo browsing and how the visual display is synchronized with gestures.
---
paper_title: Content-aware auto-soundtracks for personal photo music slideshows

paper_content:

We present a novel slideshow generation concept based on content-aware photo-music mapping. Current technologies for automated personal photo slideshow generation primarily focus on photo presentations and visual effects. These solutions utilize either a manually chosen single song or preset slideshow songs. Our technology focuses on an automatic soundtrack generation process that attempts to comprehend what the photos depict and choose music accordingly. The process comprises analyzing a photo album, composing an auto-soundtrack by dynamically mapping song segments to photo events, and automatically choosing content-aware transitions and visual effects appropriate for the presentation.
---
paper_title: Image integrity and aesthetics: towards a more encompassing definition of visual quality

paper_content:

Visual quality is a multifaceted quantity that depends on multiple attributes of the image/video. According to Keelan's ::: definition, artifactual attributes concern features of the image that when visible, are annoying and compromise the ::: integrity of the image. Aesthetic attributes instead depend on the observer's personal taste. Both types of attributes have ::: been studied in the literature in relation to visual quality, but never in conjunction with each other. In this paper we ::: perform a psychometric experiment to investigate how artifactual and aesthetic attributes interact, and how they affect ::: the viewing behavior. In particular, we studied to what extent the appearance of artifacts impacts the aesthetic quality of ::: images. Our results indicate that indeed image integrity somehow influences the aesthetic quality scores. By means of an ::: eye-tracker, we also recorded and analyzed the viewing behavior of our participants while scoring aesthetic quality. ::: Results reveal that, when scoring aesthetic quality, viewing behavior significantly departs from the natural free looking, ::: as well as from the viewing behavior observed for integrity scoring.
---
paper_title: Supporting personal photo storytelling for social albums

paper_content:

Information overload is one of today's major concerns. As high-resolution digital cameras become increasingly pervasive, unprecedented amounts of social media are being uploaded to online social networks on a daily basis. In order to support users on selecting the best photos to create an online photo album, attention has been devoted to the development of automatic approaches for photo storytelling. In this paper, we present a novel photo collection summarization system that learns some of the users' social context by analyzing their online photo albums, and includes storytelling principles and face and image aesthetic ranking in order to assist users in creating new photo albums to be shared online. In an in-depth user study conducted with 12 subjects, the proposed system was validated as a first step in the photo album creation process, helping users reduce workload to accomplish such a task. Our findings suggest that a human audio/video professional with cinematographic skills does not perform better than our proposed system.
---
paper_title: Consumer photo management and browsing facilitated by near-duplicate detection with feature filtering

paper_content:

Near-duplicate detection techniques are exploited to facilitate representative photo selection and region-of-interest (ROI) determination, which are important functionalities for efficient photo management and browsing. To make near-duplicate detection module resist to noisy features, three filtering approaches, i.e., point-based, region-based, and probabilistic latent semantic (pLSA), are developed to categorize feature points. For the photos taken in travels, we construct a support vector machine classifier to model matching patterns between photos and determine whether photos are near-duplicate pairs. Relationships between photos are then described as a graph, and the most central photo that best represents a photo cluster is selected according to centrality values. Because matched feature points are often located in the interior or at the contour of important objects, the region that compactly covers the matched feature points is determined as the ROI. We compare the proposed approaches with conventional ones and demonstrate their effectiveness.
---
paper_title: Summarizing tourist destinations by mining user-generated travelogues and photos

paper_content:

Automatically summarizing tourist destinations with both textual and visual descriptions is highly desired for online services such as travel planning, to facilitate users to understand the local characteristics of tourist destinations. Travelers are contributing a great deal of user-generated travelogues and photos on the Web, which contain abundant travel-related information and cover various aspects (e.g., landmarks, styles, activities) of most locations in the world. To leverage the collective knowledge of travelers for destination summarization, in this paper we propose a framework which discovers location-representative tags from travelogues and then select relevant and representative photos to visualize these tags. The learnt tags and selected photos are finally organized appropriately to provide an informative summary which describes a given destination both textually and visually. Experimental results based on a large collection of travelogues and photos show promising results on destination summarization.
---
paper_title: PhotoCube: effective and efficient multi-dimensional browsing of personal photo collections

paper_content:

It has never been so easy to take pictures, and personal image collections have never been so large. Unfortunately, most current photo browsers provide very limited support for effectively navigating image collections. This demonstration proposal describes PhotoCube, a personal image browser based on a multi-dimensional data model similar to the model used in OLAP applications. With PhotoCube, users can tag pictures, structure tags into various hierarchies, and browse images according to any possible perspective. We also describe three demonstration scenarios that show the power, flexibility and scalability of PhotoCube.
---
paper_title: An online noise filter for eye-tracker data recorded in a virtual environment

paper_content:

A Recursive Online Weight Average filter (ROWA) is proposed to remove and replace noisy data obtained from eye tracker. Since the filter can be implemented online, it can detect and replace noisy data using solely past records. Simulations results indicate that the filter achieved the same performance compared to other standard offline filters while being simpler.
---
paper_title: On-the-fly rounding (computing arithmetic)

paper_content:

In implementations of operations based on digit-recurrence algorithms such as division, left-to-right multiplication and square root, the result is obtained in digit-serial form, from most significant digit to least significant. To reduce the complexity of the result-digit selection and allow the use of redundant addition, the result-digit has values from a signed-digit set. As a consequence, the result has to be converted to conventional representation, which can be done on-the-fly as the digits are produced, without the use of a carry-propagate adder. The authors describe three ways to modify this conversion process so that the result is rounded. The resulting operation is fast because no carry-propagate addition is needed. The schemes described apply also to online arithmetic operations. >
---
paper_title: Stereo vision using two PTZ cameras

paper_content:

The research of traditional stereo vision is mainly based on static cameras. As PTZ (Pan-Tilt-Zoom) cameras are able to obtain multi-view-angle and multi-resolution information, they have received more and more concern in both research and real application. Stereo vision using dual-PTZ-camera system, compared with using dual-static-camera system, is much more challenging. Dual-PTZ-camera system could have more extensive scope of application by combining the merits of PTZ-camera. However, few works about stereo vision with dual-PTZ-camera system were found in literature. In this paper, we propose a novel stereo rectification method for dual-PTZ-camera system, which is essential to greatly increase the efficiency of stereo matching. In dual-PTZ-camera system, the inconsistence of intensities in two camera images, which is caused by camera's self-adjustment of intensity under different illumination condition with different view fields, is also a challenge in stereo matching. In order to deal with this problem, we propose a two-step based stereo matching strategy. Experimental results show that our approach works well.
---
paper_title: Say Cheese! Experiences with a Robot Photographer

paper_content:

We have developed an autonomous robot system that takes well-composed photographs of people at social events, such as weddings and conference receptions. The robot, Lewis, navigates through the environment, opportunistically taking photographs of people. In this article, we outline the overall architecture of the system and describe how the various components interrelate. We also describe our experiences deploying the robot photographer at a number of real-world events.
---
paper_title: Unsupervised automation of photographic composition rules in digital still cameras

paper_content:

When taking pictures, professional photographers apply photographic composition rules, e.g. rule of thirds. The rule of thirds says to place the main subject's center at one of four places: at 1/3 or 2/3 of the picture width from left edge, and 1/3 or 2/3 of the picture height from the top edge. This paper develops low-complexity unsupervised methods for digital still cameras to (1) segment the main subject and (2) realize the rule-of-thirds. ::: ::: The main subject segmentation method uses the auto-focus filter, opens the shutter aperture fully, and segments the resulting image. These camera settings place the main subject in focus and blur the rest of the image by diffused light. The segmentation utilizes the difference in frequency content between the main subject and blurred background. The segmentation does not depend on prior knowledge of the indoor/outdoor setting or scene content. ::: ::: The rule-of-thirds method moves the centroid of the main subject to the closest of the four rule-of-thirds locations. We first define an objective function that measures how close the main subject placement obeys the rule-of-thirds, and then reposition the main subject in order to optimize the objective function. For multiple main subjects, the proposed algorithm could be extended to use rule-of-triangles by adding an appropriate constraint.
---
paper_title: An autonomous robot photographer

paper_content:

We describe a complete, end-to-end system for taking well-composed photographs using a mobile robot. The general scenario is a reception, or other event, where people are roaming around talking to each other. The robot serves as an "event photographer", roaming around the same space as the participants, periodically taking photographs. These images are then sent to a workstation where participants can print the photographs out, or email them.
---
paper_title: Real-time Face-based Auto-Focus for Digital Still and Cell-Phone Cameras

paper_content:

Auto-focus (AF) is a common feature in consumer level digital and cell-phone cameras. Face-based AF, or AF based on face detection, has become of interest due to the fact that the majority of pictures captured by consumers are of human faces. While many face detection algorithms exist in the literature, very few of them are actually suitable for real-time deployment on resource limited digital or cell-phone camera processors. In this paper, a face-detection algorithm combining a Gaussian skin color model with a computationally efficient sub-block postprocessing scheme is introduced to address the realtime constraints encountered in digital and cellphone cameras. This approach has been implemented in conjunction with our previously developed rule- based AF method in order to achieve real-time faced- based AF on the Texas Instruments programmable TMS320DM350 digital camera processor.
---
paper_title: A framework for photo-quality assessment and enhancement based on visual aesthetics

paper_content:

We present an interactive application that enables users to improve the visual aesthetics of their digital photographs using spatial recomposition. Unlike earlier work that focuses either on photo quality assessment or interactive tools for photo editing, we enable the user to make informed decisions about improving the composition of a photograph and to implement them in a single framework. Specifically, the user interactively selects a foreground object and the system presents recommendations for where it can be moved in a manner that optimizes a learned aesthetic metric while obeying semantic constraints. For photographic compositions that lack a distinct foreground object, our tool provides the user with cropping or expanding recommendations that improve its aesthetic quality. We learn a support vector regression model for capturing image aesthetics from user data and seek to optimize this metric during recomposition. Rather than prescribing a fully-automated solution, we allow user-guided object segmentation and inpainting to ensure that the final photograph matches the user's criteria. Our approach achieves 86% accuracy in predicting the attractiveness of unrated images, when compared to their respective human rankings. Additionally, 73% of the images recomposited using our tool are ranked more attractive than their original counterparts by human raters.
---
paper_title: Content-aware photo-on-photo composition for consumer photos

paper_content:

We propose an approach to automatically generate photo-on-photo compositions using only a consumer's own image collection, without imposing a strong theme constraint dictated by typical background art designs. Our goal is to ensure adequate foreground and background photo sizing, avoid occlusion of important background photo content, and respect simple aesthetic rules in photo arrangement. We propose a novel image content redundancy measure to extract the informative zone, which is a rectangular region of the background photo that should not be occluded by foreground content. Background photo layout and foreground layout regions are then calculated based on a placement algorithm that deals with possible aspect ratio mismatch of the canvas and background photo, as well as cases of insufficient foreground photo layout area.
---
paper_title: Automatic local contrast enhancement using adaptive histogram adjustment

paper_content:

This paper presents the adaptive histogram adjustment (AHA) to improve image contrast. The proposed method is achieved based on the concepts of weighted histogram separation and gray-level grouping. It not only improves the contrast of the local detail but also solves group density and blocky effect which occur at the under-quantization problem of weighted histogram separation. Moreover, the adaptive histogram adjustment can prevent the contrast over-enhancement of conventional adaptive-based approach. The experimental results show that AHA has good contrast sensitivity, and it is compared with the five existing approaches, such as, histogram equalization, adaptive histogram equalization, weighted histogram separation, adaptive weighted histogram separation and gray-level grouping.
---
paper_title: Automatic thumbnail cropping and its effectiveness

paper_content:

Thumbnail images provide users of image retrieval and browsing systems with a method for quickly scanning large numbers of images. Recognizing the objects in an image is important in many retrieval tasks, but thumbnails generated by shrinking the original image often render objects illegible. We study the ability of computer vision systems to detect key components of images so that automated cropping, prior to shrinking, can render objects more recognizable. We evaluate automatic cropping techniques 1) based on a general method that detects salient portions of images, and 2) based on automatic face detection. Our user study shows that these methods result in small thumbnails that are substantially more recognizable and easier to find in the context of visual search.
---
paper_title: Removal of Image Artifacts Due to Sensor Dust

paper_content:

Image artifacts that result from sensor dust are a common but annoying problem for many photographers. To reduce the appearance of dust in an image, we first formulate a model of artifact formation due to sensor dust. With this artifact formation model, we make use of contextual information in the image and a color consistency constraint on dust to remove these artifacts. When multiple images are available from the same camera, even under different camera settings, this approach can also be used to reliably detect dust regions on the sensor. In contrast to image inpainting or other hole-filling methods, the proposed technique utilizes image information within a dust region to guide the use of contextual data. Joint use of these multiple cues leads to image recovery results that are not only visually pleasing, but also faithful to the actual scene. The effectiveness of this method is demonstrated in experiments with various cameras.
---
paper_title: Animal detection in natural scenes: Critical features revisited

paper_content:

S. J. Thorpe, D. Fize, and C. Marlot (1996) showed how rapidly observers can detect animals in images of natural scenes, but it is still unclear which image features support this rapid detection. A. B. Torralba and A. Oliva (2003) suggested that a simple image statistic based on the power spectrum allows the absence or presence of objects in natural scenes to be predicted. We tested whether human observers make use of power spectral differences between image categories when detecting animals in natural scenes. In Experiments 1 and 2 we found performance to be essentially independent of the power spectrum. Computational analysis revealed that the ease of classification correlates with the proposed spectral cue without being caused by it. This result is consistent with the hypothesis that in commercial stock photo databases a majority of animal images are pre-segmented from the background by the photographers and this pre-segmentation causes the power spectral differences between image categories and may, furthermore, help rapid animal detection. Data from a third experiment are consistent with this hypothesis. Together, our results make it exceedingly unlikely that human observers make use of power spectral differences between animaland no-animal images during rapid animal detection. In addition, our results point to potential confounds in the commercially available “natural image” databases whose statistics may be less natural than commonly presumed.
---
paper_title: Using Human Visual System modeling for bio-inspired low level image processing

paper_content:

An efficient modeling of the processing occurring at retina level and in the V1 visual cortex has been proposed in [1,2]. The aim of the paper is to show the advantages of using such a modeling in order to develop efficient and fast bio-inspired modules for low level image processing. At the retina level, a spatio-temporal filtering ensures accurate structuring of video data (noise and illumination variation removal, static and dynamic contour enhancement). In the V1 cortex, a frequency and orientation based analysis is performed. The combined use of retina and V1 cortex modeling allows the development of low level image processing modules for contour enhancement, for moving contour extraction, for motion analysis and for motion event detection. Each module is described and its performances are evaluated. The retina model has been integrated into a real-time C/C++ optimized program which is also presented in this paper with the derived computer vision tools.
---
paper_title: Optimizing photo composition

paper_content:

Aesthetic images evoke an emotional response that transcends mere visual appreciation. In this work we develop a novel computational means for evaluating the composition aesthetics of a given image based on measuring several well-grounded composition guidelines. A compound operator of crop-and-retarget is employed to change the relative position of salient regions in the image and thus to modify the composition aesthetics of the image. We propose an optimization method for automatically producing a maximally-aesthetic version of the input image. We validate the performance of the method and show its effectiveness in a variety of experiments.
---
paper_title: Image retargeting based on global energy optimization

paper_content:

This paper proposes a novel image retargeting technique based on global energy optimization. Most existing methods enhance the high energy parts of the original image by pre-defined strategies or local optimization based iterations. They can not achieve the global optimal effect in energy retainment. To solve this problem, our approach formulates image retargeting as a global optimization problem on energy. We first calculate the energy map of the original image. Then, we utilize a constrained linear programming to maximize the retained energy in retargeting. Finally, we propose a pixel fusion based method to generate the retargeted image. To make it more feasible in implementation, we further provide two strategies to reduce the time cost of our approach. We demonstrate the proposed approach by comparing with typical image retargeting methods.
---
paper_title: A visual attention model for adapting images on small displays

paper_content:

Image adaptation, one of the essential problems in adaptive content delivery for universal access, has been actively explored for some time. Most existing approaches have focused on generic adaptation with a view to saving file size under constraints in client environment and have hardly paid attention to user perceptions of the adapted result. Meanwhile, the major limitation on the user's delivery context is moving away from data volume (or time-to-wait) to screen size because of the galloping development of hardware technologies. In this paper, we propose a novel method for adapting images based on user attention. A generic and extensible image attention model is introduced based on three attributes (region of interest, attention value, and minimal perceptible size) associated with each attention object. A set of automatic modeling methods are presented to support this approach. A branch-and-bound algorithm is also developed to find the optimal adaptation efficiently. Experimental results demonstrate the usefulness of the proposed scheme and its potential application in the future.
---
paper_title: Image Inpainting by Patch Propagation Using Patch Sparsity

paper_content:

This paper introduces a novel examplar-based inpainting algorithm through investigating the sparsity of natural image patches. Two novel concepts of sparsity at the patch level are proposed for modeling the patch priority and patch representation, which are two crucial steps for patch propagation in the examplar-based inpainting approach. First, patch structure sparsity is designed to measure the confidence of a patch located at the image structure (e.g., the edge or corner) by the sparseness of its nonzero similarities to the neighboring patches. The patch with larger structure sparsity will be assigned higher priority for further inpainting. Second, it is assumed that the patch to be filled can be represented by the sparse linear combination of candidate patches under the local patch consistency constraint in a framework of sparse representation. Compared with the traditional examplar-based inpainting approach, structure sparsity enables better discrimination of structure and texture, and the patch sparse representation forces the newly inpainted regions to be sharp and consistent with the surrounding textures. Experiments on synthetic and natural images show the advantages of the proposed approach.
---
paper_title: Learning Probabilistic Models for Contour Completion in Natural Images

paper_content:

Using a large set of human segmented natural images, we study the statistics of region boundaries. We observe several power law distributions which likely arise from both multi-scale structure within individual objects and from arbitrary viewing distance. Accordingly, we develop a scale-invariant representation of images from the bottom up, using a piecewise linear approximation of contours and constrained Delaunay triangulation to complete gaps. We model curvilinear grouping on top of this graphical/geometric structure using a conditional random field to capture the statistics of continuity and different junction types. Quantitative evaluations on several large datasets show that our contour grouping algorithm consistently dominates and significantly improves on local edge detection.
---
paper_title: Seam carving for content-aware image resizing

paper_content:

Effective resizing of images should not only use geometric constraints, but consider the image content as well. We present a simple image operator called seam carving that supports content-aware image resizing for both reduction and expansion. A seam is an optimal 8-connected path of pixels on a single image from top to bottom, or left to right, where optimality is defined by an image energy function. By repeatedly carving out or inserting seams in one direction we can change the aspect ratio of an image. By applying these operators in both directions we can retarget the image to a new size. The selection and order of seams protect the content of the image, as defined by the energy function. Seam carving can also be used for image content enhancement and object removal. We support various visual saliency measures for defining the energy of an image, and can also include user input to guide the process. By storing the order of seams in an image we create multi-size images, that are able to continuously change in real time to fit a given size.
---
paper_title: Automatic creation of face composite images for consumer applications

paper_content:

This paper presents a method to perform automatic extraction, manipulation and blending of human face and hair images into other media such as licensed commercial content. We describe automatic image processing algorithms to perform such tasks, and discuss applications in personalized publishing and merchandising. We demonstrate our work through automatically segmenting out the face and hair image in a consumer photo and embedding it in another photo or document. The goal is to perform these steps seamlessly with natural look and minimal manual intervention. The primary application of this technology is the personalization of various documents and products such as licensed merchandise, children's storybook, puzzles and marketing collaterals.
---
paper_title: Personal photo enhancement using example images

paper_content:

We describe a framework for improving the quality of personal photos by using a person's favorite photographs as examples. We observe that the majority of a person's photographs include the faces of a photographer's family and friends and often the errors in these photographs are the most disconcerting. We focus on correcting these types of images and use common faces across images to automatically perform both global and face-specific corrections. Our system achieves this by using face detection to align faces between “good” and “bad” photos such that properties of the good examples can be used to correct a bad photo. These “personal” photos provide strong guidance for a number of operations and, as a result, enable a number of high-quality image processing operations. We illustrate the power and generality of our approach by presenting a novel deblurring algorithm, and we show corrections that perform sharpening, superresolution, in-painting of over- and underexposured regions, and white-balancing.
---
paper_title: Semantic adaptation of consumer photo for mobile device access

paper_content:

This paper proposes a novel event-aware semantic image adaptation framework for accessing consumer photos via mobile devices. There are two major challenges to develop such an image adaptation framework: (1) how to bridge the semantic gap in the search for the desired photos from a consumer database; (2) how to perform appropriate adaptation for various small displays in order to maximize user's perceptual experience according to the semantic significance of objects in photos. To meet the first challenge, an event semantics-guided extraction is developed based on statistical fusion of bottom-up low level features and top-down semantic features. To meet the second challenge, a key object-based semantic adaptation approach is designed to obtain the perceptually optimized Region of Interest (ROI). We conducted experiments based on the events defined in the Kodak consumer photo database [2]. These experiments show that by utilizing the semantics of the photos with event-based a priori knowledge, the adaptation result outperforms the conventional attention based scheme. More importantly, the proposed framework can be easily extended to other events by adopting the event semantics based a priori knowledge into the adaptation system.
---
paper_title: Auto cropping for digital photographs

paper_content:

In this paper, we propose an effective approach to the nearly untouched problem, still photograph auto cropping, which is one of the important features to automatically enhance photographs. To obtain an optimal result, we first formulate auto cropping as an optimization problem by defining an energy function, which consists of three sub models: composition sub model, conservative sub model, and penalty sub model. Then, particle swarm optimization (PSO) is employed to obtain the optimal solution by maximizing the objective function. Experimental results and user studies over hundreds of photographs show that the proposed approach is effective and accurate in most cases, and can be used in many practical multimedia applications.
---
paper_title: Learning based thumbnail cropping

paper_content:

Thumbnail cropping helps improve thumbnail readability by cropping images before shrinking them. In this paper we propose a learning based method for automatic thumbnail cropping. To this end, we use a support vector machine to learn a discriminative model that simultaneously captures the saliency distribution and spatial priors. The model is then used to determine the best cropping rectangle. The proposed approach improves traditional saliency based cropping techniques by introducing the spatial priors, which is automatically learned through learning process. The new method is tested on images from the PASCAL08 dataset, where it outperforms previous saliency based cropping.
---
paper_title: A framework for photo-quality assessment and enhancement based on visual aesthetics

paper_content:

We present an interactive application that enables users to improve the visual aesthetics of their digital photographs using spatial recomposition. Unlike earlier work that focuses either on photo quality assessment or interactive tools for photo editing, we enable the user to make informed decisions about improving the composition of a photograph and to implement them in a single framework. Specifically, the user interactively selects a foreground object and the system presents recommendations for where it can be moved in a manner that optimizes a learned aesthetic metric while obeying semantic constraints. For photographic compositions that lack a distinct foreground object, our tool provides the user with cropping or expanding recommendations that improve its aesthetic quality. We learn a support vector regression model for capturing image aesthetics from user data and seek to optimize this metric during recomposition. Rather than prescribing a fully-automated solution, we allow user-guided object segmentation and inpainting to ensure that the final photograph matches the user's criteria. Our approach achieves 86% accuracy in predicting the attractiveness of unrated images, when compared to their respective human rankings. Additionally, 73% of the images recomposited using our tool are ranked more attractive than their original counterparts by human raters.
---
paper_title: Evaluating visual aesthetics in photographic portraiture

paper_content:

We propose and demonstrate a strategy to quantify aesthetic quality in photographs. Our approach is to develop a small set of classification features by tuning general compositional principles to a targeted image domain where saliency can be better understood. We demonstrate this strategy with photographic portraits of individuals, but it can be extended to other domains. Our technique leverages a refined method of using templates as spatial composition feature look-up tables. Compared to the traditional approach using a large set of global and local features extracted with little salient knowledge, classifiers using features extracted with our approach are better predictors of human aesthetic judgments.
---
paper_title: Photo and Video Quality Evaluation: Focusing on the Subject

paper_content:

Traditionally, distinguishing between high quality professional photos and low quality amateurish photos is a human task. To automatically assess the quality of a photo that is consistent with humans perception is a challenging topic in computer vision. Various differences exist between photos taken by professionals and amateurs because of the use of photography techniques. Previous methods mainly use features extracted from the entire image. In this paper, based on professional photography techniques, we first extract the subject region from a photo, and then formulate a number of high-level semantic features based on this subject and background division. We test our features on a large and diverse photo database, and compare our method with the state of the art. Our method performs significantly better with a classification rate of 93% versus 72% by the best existing method. In addition, we conduct the first study on high-level video quality assessment. Our system achieves a precision of over 95% in a reasonable recall rate for both photo and video assessments. We also show excellent application results in web image search re-ranking.
---
paper_title: Image harmony for consumer images

paper_content:

Images elicit a variety of emotional responses related to image content, overall aesthetic appeal, or a combination of both. One aspect of aesthetic appeal is harmony: the pleasing or congruent arrangement of parts producing internal calm or tranquility. We conducted a series of experiments to identify what low level features could predict harmony in an image. Subjective judgments of image harmony were collected for images representative of typical consumer photography. Our initial results show that for simplified images (pixelated to control for emotional responses to scenes and objects) impressions of image harmony depend on statistical properties of low level local features. The feature combinations vary for different individuals but typically involve edge contrast, average lightness and range of lightness. At the same time inclusion of Gestalt principles is needed to account for the subjects' data. Additionally, global low level features related to spatial image structure may help to explain results with black and white and color images.
---
paper_title: Say Cheese! Experiences with a Robot Photographer

paper_content:

We have developed an autonomous robot system that takes well-composed photographs of people at social events, such as weddings and conference receptions. The robot, Lewis, navigates through the environment, opportunistically taking photographs of people. In this article, we outline the overall architecture of the system and describe how the various components interrelate. We also describe our experiences deploying the robot photographer at a number of real-world events.
---
paper_title: Towards intelligent photo composition-automatic detection of unintentional dissection lines in environmental portrait photos

paper_content:

Good photos result from careful attention to various elements such as shutter speed, aperture, light exposure and composition. Over the years, there have been interests to develop digital still cameras with ''composition advising'' functions. However, not many composition-related functions have been realized in current digital cameras. This study proposed an approach for automatic detection of unintentional dissection lines which often degrade the aesthetics of an environmental portrait photo. The algorithm includes modules for face detection, ROI estimation, morphology filtering, edge detection and straight line detection. Experimental evaluations conducted to verify the performance of the algorithm show that the detection rate and false detection rate of general dissection lines are 80.87% and 33.61%, respectively.
---
paper_title: Optimizing photo composition

paper_content:

Aesthetic images evoke an emotional response that transcends mere visual appreciation. In this work we develop a novel computational means for evaluating the composition aesthetics of a given image based on measuring several well-grounded composition guidelines. A compound operator of crop-and-retarget is employed to change the relative position of salient regions in the image and thus to modify the composition aesthetics of the image. We propose an optimization method for automatically producing a maximally-aesthetic version of the input image. We validate the performance of the method and show its effectiveness in a variety of experiments.
---
paper_title: Unsupervised automation of photographic composition rules in digital still cameras

paper_content:

When taking pictures, professional photographers apply photographic composition rules, e.g. rule of thirds. The rule of thirds says to place the main subject's center at one of four places: at 1/3 or 2/3 of the picture width from left edge, and 1/3 or 2/3 of the picture height from the top edge. This paper develops low-complexity unsupervised methods for digital still cameras to (1) segment the main subject and (2) realize the rule-of-thirds. ::: ::: The main subject segmentation method uses the auto-focus filter, opens the shutter aperture fully, and segments the resulting image. These camera settings place the main subject in focus and blur the rest of the image by diffused light. The segmentation utilizes the difference in frequency content between the main subject and blurred background. The segmentation does not depend on prior knowledge of the indoor/outdoor setting or scene content. ::: ::: The rule-of-thirds method moves the centroid of the main subject to the closest of the four rule-of-thirds locations. We first define an objective function that measures how close the main subject placement obeys the rule-of-thirds, and then reposition the main subject in order to optimize the objective function. For multiple main subjects, the proposed algorithm could be extended to use rule-of-triangles by adding an appropriate constraint.
---
paper_title: Support-Vector Networks

paper_content:

The support-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data. ::: ::: High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: The Design of High-Level Features for Photo Quality Assessment

paper_content:

We propose a principled method for designing high level features forphoto quality assessment. Our resulting system can classify between high quality professional photos and low quality snapshots. Instead of using the bag of low-level features approach, we first determine the perceptual factors that distinguish between professional photos and snapshots. Then, we design high level semantic features to measure the perceptual differences. We test our features on a large and diverse dataset and our system is able to achieve a classification rate of 72% on this difficult task. Since our system is able to achieve a precision of over 90% in low recall scenarios, we show excellent results in a web image search application.
---
paper_title: ACQUINE: aesthetic quality inference engine - real-time automatic rating of photo aesthetics

paper_content:

We present ACQUINE - Aesthetic Quality Inference Engine, a publicly accessible system which allows users to upload their photographs and have them rated automatically for aesthetic quality. The system integrates a support vector machine based classifier which extracts visual features on the fly and performs real-time classification and prediction. As the first publicly available tool for automatically determining the aesthetic value of an image, this work is a significant first step in recognizing human emotional reaction to visual stimulus. In this paper, we discuss fundamentals behind this system, and some of the challenges faced while creating it. We report statistics generated from over 140,000 images uploaded by Web users. The system is demonstrated at http://acquine.alipr.com.
---
paper_title: On confidence and response times of human observers in subjective image quality assessment

paper_content:

Mean opinion scores obtained in subjective image quality experiments are widely accepted as measures of perceived visual quality. They have, however, a strong limitation regarding the reliability of the rated quality, since there is no explicit information as to whether the human observer experienced difficulties when judging image quality. We thus suggest that additional information about the observers confidence should be provided along with the actual quality measure. In this paper, we analyse two ways of obtaining this confidence measure; firstly as a confidence score given by the human observer and secondly as an indirect measure of the observers response time to provide the quality score. We reveal strong relationships of confidence scores and response times to the quality scores. We further propose a model to predict observer confidence based on the quality scores and response times.
---
paper_title: Emotional valence categorization using holistic image features

paper_content:

Can a machine learn to perceive emotions as evoked by an artwork? Here we propose an emotion categorization system, trained by ground truth from psychology studies. The training data contains emotional valences scored by human subjects on the International Affective Picture System (IAPS), a standard emotion evoking image set in psychology. Our approach is based on the assessment of local image statistics which are learned per emotional category using support vector machines. We show results for our system on the I APS dataset, and for a collection of masterpieces. Although the results are preliminary, they demonstrate the potential of machines to elicit realistic emotions when considering masterpieces.
---
paper_title: Towards computational models of the visual aesthetic appeal of consumer videos

paper_content:

In this paper, we tackle the problem of characterizing the aesthetic appeal of consumer videos and automatically classifying them into high or low aesthetic appeal. First, we conduct a controlled user study to collect ratings on the aesthetic value of 160 consumer videos. Next, we propose and evaluate a set of low level features that are combined in a hierarchical way in order to model the aesthetic appeal of consumer videos. After selecting the 7 most discriminative features, we successfully classify aesthetically appealing vs. aesthetically unappealing videos with a 73% classification accuracy using a support vector machine.
---
paper_title: Evaluation of image appeal in consumer photography

paper_content:

In consumer photography, image appeal may be defined by the interest that a picture generates when viewed by third-party observers. In this paper, the results of a ground truth experiment on human estimation of image appeal are reported, where 11 participants were asked to rank pictures in 30 groups, based on their relative appeal within their group and comment on the factors that influenced their decisions. Based on their responses, a list of both positive and negative influences was compiled and influences were grouped in general categories that include people, composition, subject, and objective metrics. The results of our experiment indicate that image appeal is related to image quality only with respect to the influences in the category of objective metrics, while the majority of influences belong to the categories of people, composition, and subject. The influences in these categories are scene dependent and fundamentally different in nature from traditional image quality metrics. Thus, when evaluating image appeal a new set of metrics needs to be developed. Individual influences and their relative merits are discussed.
---
paper_title: Algorithmic inferencing of aesthetics and emotion in natural images: An exposition

paper_content:

Initial studies have shown that automatic inference of high-level image quality or aesthetics is very challenging. The ability to do so, however, can prove beneficial in many applications. In this paper, we define the aesthetics gap and discuss key aspects of the problem of aesthetics and emotion inference in natural images. We introduce precise, relevant questions to be answered, the effect that the target audience has on the problem specification, broad technical solution approaches, and assessment criteria. We then report on our effort to build real-world datasets that provide viable approaches to test and compare algorithms for these problems, presenting statistical analysis of and insights into them.
---
paper_title: Towards aesthetics: a photo quality assessment and photo selection system

paper_content:

Automatic photo quality assessment and selection systems are helpful for managing the large mount of consumer photos. In this paper, we present such a system based on evaluating the aesthetic quality of consumer photos. The proposed system focuses on photos with faces, which constitute an important part of consumer photo albums. The system has three contributions: 1) We propose an aesthetics-based photo assessment algorithm, by considering different aesthetics-related factors, including the technical characteristics of the photo and the specific features related to faces; 2) Based on the aesthetic measurement, we propose a cropping-based photo editing algorithm, which differs from prior works by eliminating unimportant faces before optimizing photo composition; 3) We also incorporate the aesthetic evaluation with other metrics to select quintessential photos for a large collection of photos. The entire system is delivered by a web interface, which allows users to submit images or albums, and returns promising results for photo evaluation, editing recommendation, and photo selection.
---
paper_title: Region based image appeal metric for consumer photos

paper_content:

Image appeal may be defined as the interest that a photograph generates when viewed by human observers, incorporating subjective factors on top of the traditional objective quality measures. User studies were conducted in order to identify the right features to use in an image appeal measure; these studies also revealed that a photograph may be appealing even if only a region/area of the photograph is actually appealing. Extensive experimentation helped identify a good set of low level features, which were combined into a set of image appeal metrics. The appealing region detection and the image appeal metrics were validated with experiments conducted on 2000 ground truth images, each graded by three human observers. Two main image appeal metrics are presented: one that ranks the images based on image appeal with a high correlation with the human observations; and a second one which performs better at retrieving highly appealing images from an image collection.
---
paper_title: Aesthetic quality assessment of consumer photos with faces

paper_content:

Automatically assessing the subjective quality of a photo is a challenging area in visual computing. Previous works study the aesthetic quality assessment on a general set of photos regardless of the photo's content and mainly use features extracted from the entire image. In this work, we focus on a specific genre of photos: consumer photos with faces. This group of photos constitutes an important part of consumer photo collections. We first conduct an online study on Mechanical Turk to collect ground-truth and subjective opinions for a database of consumer photos with faces. We then extract technical features, perceptual features, and social relationship features to represent the aesthetic quality of a photo, by focusing on face-related regions. Experiments show that our features perform well for categorizing or predicting the aesthetic quality.
---
paper_title: Personalized photograph ranking and selection system

paper_content:

In this paper, we propose a novel personalized ranking system for amateur photographs. Although some of the features used in our system are similar to previous work, new features, such as texture, RGB color, portrait (through face detection), and black-and-white, are included for individual preferences. Our goal of automatically ranking photographs is not intended for award-wining professional photographs but for photographs taken by amateurs, especially when individual preference is taken into account. The performance of our system in terms of precision-recall diagram and binary classification accuracy (93%) is close to the best results to date for both overall system and individual features. Two personalized ranking user interfaces are provided: one is feature-based and the other is example-based. Although both interfaces are effective in providing personalized preferences, our user study showed that example-based was preferred by twice as many people as feature-based.
---
paper_title: Using Group Prior to Identify People in Consumer Images

paper_content:

While face recognition techniques have rapidly advanced in the last few years, most of the work is in the domain of security applications. For consumer imaging applications, person recognition is an important tool that is useful for searching and retrieving images from a personal image collection. It has been shown that when recognizing a single person in an image, a maximum likelihood classifier requires the prior probability for each candidate individual. In this paper, we extend this idea and describe the benefits of using a group prior for identifying people in consumer images with multiple people. The group prior describes the probability of a group of individuals appearing together in an image. In our application, we have a subset of ambiguously labeled images for a consumer image collection, where we seek to identify all of the people in the collection. We describe a simple algorithm for resolving the ambiguous labels. We show that despite errors in resolving ambiguous labels, useful classifiers can be trained with the resolved labels. Recognition performance is further improved with a group prior learned from the ambiguous labels. In summary, by modeling the relationships between the people with the group prior, we improve classification performance.
---
paper_title: Induction of decision trees

paper_content:

The technology for building knowledge-based systems by inductive inference from examples has been demonstrated successfully in several practical applications. This paper summarizes an approach to synthesizing decision trees that has been used in a variety of systems, and it describes one such system, ID3, in detail. Results from recent studies show ways in which the methodology can be modified to deal with information that is noisy and/or incomplete. A reported shortcoming of the basic algorithm is discussed and two means of overcoming it are compared. The paper concludes with illustrations of current research directions.
---
paper_title: Hierarchical pose classification based on human physiology for behaviour analysis

paper_content:

This study presents a new approach to classify human body poses by using angular constraints and variations of body joints. Although different classifications of the poses have been previously made, the proposed approach attempts to create a more comprehensive, accurate and extensible classification by integrating all possible poses based on angles of movement in human joints. The angular variations in all body joints can determine any possible poses. The joint angles from the body axis are computed in the three-dimensional space. In order to train and classify the pose in an automated manner, support vector machines (SVMs) were used. Experiments were carried out on both benchmark (CMU dataset) and in-house simulated (POSER dataset) poses to evaluate the performance of the proposed classification scheme.
---
paper_title: Skin Patch Detection in Real-World Images

paper_content:

While human skin is relatively easy to detect in controlled environments, detection in uncontrolled settings such as in consumer digital photographs is generally hard. Algorithms need to robustly deal with variations in lighting, color resolution, and imaging noise. This paper proposes a simple generative skin patch model combining shape and color information. The model is parametric and represents the spatial arrangement of skin pixels as compact elliptical regions. Its parameters are estimated by maximizing the mutual information between the model-generated skin pixel distribution and the distribution of skin color as observed in the image. The core of this work is an empirical evaluation on a database of 653 consumer digital photographs. In addition, we investigate the potential of combining our skin detector with state-of-the-art appearance-based face detectors.
---
paper_title: Photo and Video Quality Evaluation: Focusing on the Subject

paper_content:

Traditionally, distinguishing between high quality professional photos and low quality amateurish photos is a human task. To automatically assess the quality of a photo that is consistent with humans perception is a challenging topic in computer vision. Various differences exist between photos taken by professionals and amateurs because of the use of photography techniques. Previous methods mainly use features extracted from the entire image. In this paper, based on professional photography techniques, we first extract the subject region from a photo, and then formulate a number of high-level semantic features based on this subject and background division. We test our features on a large and diverse photo database, and compare our method with the state of the art. Our method performs significantly better with a classification rate of 93% versus 72% by the best existing method. In addition, we conduct the first study on high-level video quality assessment. Our system achieves a precision of over 95% in a reasonable recall rate for both photo and video assessments. We also show excellent application results in web image search re-ranking.
---
paper_title: Classification of silhouettes using contour fragments

paper_content:

In this paper, we propose a fragment-based approach for classification and recognition of shape contours. According to this method, first the perceptual landmarks along the contours are localized in a scale invariant manner, which makes it possible to extracts the contour fragments. Using a predefined dictionary for the fragments, these landmarks and the parts between them are transformed into a symbolic representation that is a compact representation. Using a string kernel-like approach, an invariant high-dimensional feature space is created from the symbolic representation and later the most relevant lower dimensions are extracted by principal component analysis. Finally, support vector machine is used for classification of the feature space. The experimental results show that the proposed method has similar performance to the best approaches for shape recognitions while it has lower complexity.
---
paper_title: Multi-label image segmentation via point-wise repetition

paper_content:

Bottom-up segmentation tends to rely on local features. Yet, many natural and man-made objects contain repeating elements. Such structural and more spread-out features are important cues for segmentation but are more difficult to exploit. The difficulty also comes from the fact that repetition need not be perfect, and will actually rather be partial, approximate, or both in most cases. This paper presents a multi-label image segmentation algorithm that processes a single input image and efficiently discovers and exploits repeating elements without any prior knowledge about their shape, color or structure. The algorithm spells out the interplay between segmentation and repetition detection. The key of our approach is a novel, point-wise concept of repetition. This is defined by point-wise mutual information and locally compares certain neighborhoods to accumulate evidence. This point-wise repetition measure naturally handles imperfect repetitions, and the parts with inconsistent appearances are recognized and assigned with low scores. An energy functional is proposed to include the point-wise repetition into the image segmentation process, which takes the form of a graph-cut minimization. Real scene images demonstrate the ability of our algorithm to handle partial and approximate repetition.
---
paper_title: Skin Detail Analysis for Face Recognition

paper_content:

This paper presents a novel framework to localize in a photograph prominent irregularities in facial skin, in particular nevi (moles, birthmarks). Their characteristic configuration over a face is used to encode the person's identity independent of pose and illumination. This approach extends conventional recognition methods, which usually disregard such small scale variations and thereby miss potentially highly discriminative features. Our system detects potential nevi with a very sensitive multi scale template matching procedure. The candidate points are filtered according to their discriminative potential, using two complementary methods. One is a novel skin segmentation scheme based on gray scale texture analysis that we developed to perform outlier detection in the face. Unlike most other skin detection/segmentation methods it does not require color input. The second is a local saliency measure to express a point's uniqueness and confidence taking the neighborhood's texture characteristics into account. We experimentally evaluate the suitability of the detected features for identification under different poses and illumination on a subset of the FERET face database.
---
paper_title: Towards intelligent photo composition-automatic detection of unintentional dissection lines in environmental portrait photos

paper_content:

Good photos result from careful attention to various elements such as shutter speed, aperture, light exposure and composition. Over the years, there have been interests to develop digital still cameras with ''composition advising'' functions. However, not many composition-related functions have been realized in current digital cameras. This study proposed an approach for automatic detection of unintentional dissection lines which often degrade the aesthetics of an environmental portrait photo. The algorithm includes modules for face detection, ROI estimation, morphology filtering, edge detection and straight line detection. Experimental evaluations conducted to verify the performance of the algorithm show that the detection rate and false detection rate of general dissection lines are 80.87% and 33.61%, respectively.
---
paper_title: People detection through quantified fuzzy temporal rules

paper_content:

The knowledge about the position and movement of people is of great importance in mobile robotics for implementing tasks such as navigation, mapping, localization, or human-robot interaction. This knowledge enhances the robustness, reliability and performance of the robot control architecture. In this paper, a pattern classifier system for the detection of people using laser range finders data is presented. The approach is based on the quantified fuzzy temporal rules (QFTRs) knowledge representation and reasoning paradigm, that is able to analyze the spatio-temporal patterns that are associated to people. The pattern classifier system is a knowledge base made up of QFTRs that were learned with an evolutionary algorithm based on the cooperative-competitive approach together with token competition. A deep experimental study with a Pioneer II robot involving a five-fold cross-validation and several runs of the genetic algorithm has been done, showing a classification rate over 80%. Moreover, the characteristics of the tests represent complex and realistic conditions (people moving in groups, the robot moving in part of the experiments, and the existence of static and moving people).
---
paper_title: Fusing Local Patterns of Gabor Magnitude and Phase for Face Recognition

paper_content:

Gabor features have been known to be effective for face recognition. However, only a few approaches utilize phase feature and they usually perform worse than those using magnitude feature. To investigate the potential of Gabor phase and its fusion with magnitude for face recognition, in this paper, we first propose local Gabor XOR patterns (LGXP), which encodes the Gabor phase by using the local XOR pattern (LXP) operator. Then, we introduce block-based Fisher's linear discriminant (BFLD) to reduce the dimensionality of the proposed descriptor and at the same time enhance its discriminative power. Finally, by using BFLD, we fuse local patterns of Gabor magnitude and phase for face recognition. We evaluate our approach on FERET and FRGC 2.0 databases. In particular, we perform comparative experimental studies of different local Gabor patterns. We also make a detailed comparison of their combinations with BFLD, as well as the fusion of different descriptors by using BFLD. Extensive experimental results verify the effectiveness of our LGXP descriptor and also show that our fusion approach outperforms most of the state-of-the-art approaches.
---
paper_title: Geometric active contours without re-initialization for image segmentation

paper_content:

A geometric active contour model without re-initialization that can be used for grey and color image segmentation is presented in this paper. It combines directional information about edge location based on Cumani operator as a part of driving force, with the improved geodesic active contours containing Bays error based statistical region information. Moreover, an extra term that penalizes the deviation of the level set function from a signed distance function is also included in the model, thus the costly re-initialization procedure can be completely eliminated and all these measures are integrated in a unified frame. Experimental results on real grey and color images have shown that our model can precisely extract contours of images and its performance is much better and faster than the geodesic-aided C-V (GACV) model.
---
paper_title: Motion blur identification from image gradients

paper_content:

Restoration of a degraded image from motion blurring is highly dependent on the estimation of the blurring kernel. Most of the existing motion deblurring techniques model the blurring kernel with a shift-invariant box filter, which holds true only if the motion among images is of uniform velocity. In this paper, we present a spectral analysis of image gradients, which leads to a better configuration for identifying the blurring kernel of more general motion types (uniform velocity motion, accelerated motion and vibration). Furthermore, we introduce a hybrid Fourier-Radon transform to estimate the parameters of the blurring kernel with improved robustness to noise over available techniques. The experiments on both simulated images and real images show that our algorithm is capable of accurately identifying the blurring kernel for a wider range of motion types.
---
paper_title: An efficient local Chan-Vese model for image segmentation

paper_content:

In this paper, a new local Chan-Vese (LCV) model is proposed for image segmentation, which is built based on the techniques of curve evolution, local statistical function and level set method. The energy functional for the proposed model consists of three terms, i.e., global term, local term and regularization term. By incorporating the local image information into the proposed model, the images with intensity inhomogeneity can be efficiently segmented. In addition, the time-consuming re-initialization step widely adopted in traditional level set methods can be avoided by introducing a new penalizing energy. To avoid the long iteration process for level set evolution, an efficient termination criterion is presented which is based on the length change of evolving curve. Particularly, we proposed constructing an extended structure tensor (EST) by adding the intensity information into the classical structure tensor for texture image segmentation. It can be found that by combining the EST with our LCV model, the texture image can be efficiently segmented no matter whether it presents intensity inhomogeneity or not. Finally, experiments on some synthetic and real images have demonstrated the efficiency and robustness of our model. Moreover, comparisons with the well-known Chan-Vese (CV) model and recent popular local binary fitting (LBF) model also show that our LCV model can segment images with few iteration times and be less sensitive to the location of initial contour and the selection of governing parameters.
---
paper_title: Sunset scene classification using simulated image recomposition

paper_content:

Knowledge of the semantic classification of an image can be used to improve the accuracy of queries in content-based image organization and retrieval and to provide customized image enhancement. We developed an exemplar-based system for classifying sunset scenes. However, the performance of such a system depends largely on the size and quality of the set of training exemplars, which can be limited in practice. In addition, variations in scene content, as well as distracting regions, may exist in many testing images to prohibit good matches with the exemplars. We propose using simulated spatial and temporal image recomposition to address such issues. The recomposition schemes boost the recall of sunset images from a reasonably large data set by 10%, while holding the false positive rate constant.
---
paper_title: A data mining approach to face detection

paper_content:

In this paper, we propose a novel face detection method based on the MAFIA algorithm. Our proposed method consists of two phases, namely, training and detection. In the training phase, we first apply Sobel's edge detection operator, morphological operator, and thresholding to each training image, and transform it into an edge image. Next, we use the MAFIA algorithm to mine the maximal frequent patterns from those edge images and obtain the positive feature pattern. Similarly, we can obtain the negative feature pattern from the complements of edge images. Based on the feature patterns mined, we construct a face detector to prune non-face candidates. In the detection phase, we apply a sliding window to the testing image in different scales. For each sliding window, if the slide window passes the face detector, it is considered as a human face. The proposed method can automatically find the feature patterns that capture most of facial features. By using the feature patterns to construct a face detector, the proposed method is robust to races, illumination, and facial expressions. The experimental results show that the proposed method has outstanding performance in the MIT-CMU dataset and comparable performance in the BioID dataset in terms of false positive and detection rate.
---
paper_title: Automatic skin decomposition based on single image

paper_content:

This paper proposes an automatic decomposition algorithm for separating skin image into melanin and hemoglobin components. First, skin region is detected from input image. Then, the whole image is divided into several sub regions. After extracting separation vectors from each sub regions, we use proposed method to distinguish the vectors which corresponds to melanin and hemoglobin components, respectively. Based on our validity rules, invalid vectors are discarded and valid vectors are treated as candidate vectors. Finally, we determine the final separation vectors which have minimal deviation from all of the candidate vectors to separate skin image. The separation results demonstrate the validity of our algorithm. Our experiments with lower resolution and poor quality images as input also verify that the proposed algorithm is very robust.
---
paper_title: Finding rows of people in group images

paper_content:

People are among the most popular subjects in photography, and in many social settings, images of groups of people are captured. People often arrange themselves in a very structured manner in these group images. For example, taller peoplemight stand in a row behind smaller people. This structure is often exploited in captions that sequentially label the individuals in each row. We present an algorithm for automatically finding rows of people in group images. A graph is defined for the image, where each face is a vertex. Energy terms are learned from a training set of images. A minimum cut on this graph defines the rows of people in the image. On our test set, the algorithm achieves perfect results with 67.5% of the images. Detecting rows of people is useful for a number of applications.
---
paper_title: Objective evaluation of approaches of skin detection using ROC analysis

paper_content:

Skin detection is an important indicator of human presence and actions in many domains, including interaction, interfaces and security. It is commonly performed in three steps: transforming the pixel color to a non-RGB colorspace, dropping the illuminance component of skin color, and classifying by modeling the skin color distribution. In this paper, we evaluate the effect of these three steps on the skin detection performance. The importance of this study is a new comprehensive colorspace and color modeling testing methodology that would allow for making the best choices for skin detection. Combinations of nine colorspaces, the presence or the absence of the illuminance component, and the two color modeling approaches are compared for different settings (indoor or outdoor) and modeling parameters (the histogram size). The performance is measured by using a receiver operating characteristic (ROC) curve on a large dataset of 845 images (consisting more than 18.6 million pixels) with manual ground truth. The results reveal that (1) colorspace transformations can improve performance in certain instances, (2) the absence of the illuminance component decreases performance, and (3) skin color modeling has a greater impact than colorspace transformation. We found that the best performance was obtained on indoor images by transforming the pixel color to the HSI or SCT colorspaces, keeping the illuminance component, and modeling the color with the histogram approach using a larger size distribution.
---
paper_title: Support-Vector Networks

paper_content:

The support-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data. ::: ::: High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: Content-Based Photo Quality Assessment

paper_content:

Automatically assessing photo quality from the perspective of visual aesthetics is of great interest in high-level vision research and has drawn much attention in recent years. In this paper, we propose content-based photo quality assessment using regional and global features. Under this framework, subject areas, which draw the most attentions of human eyes, are first extracted. Then regional features extracted from subject areas and the background regions are combined with global features to assess the photo quality. Since professional photographers may adopt different photographic techniques and may have different aesthetic criteria in mind when taking different types of photos (e.g. landscape versus portrait), we propose to segment regions and extract visual features in different ways according to the categorization of photo content. Therefore we divide the photos into seven categories based on their content and develop a set of new subject area extraction methods and new visual features, which are specially designed for different categories. This argument is supported by extensive experimental comparisons of existing photo quality assessment approaches as well as our new regional and global features over different categories of photos. Our new features significantly outperform the state-of-the-art methods. Another contribution of this work is to construct a large and diversified benchmark database for the research of photo quality assessment. It includes 17, 613 photos with manually labeled ground truth.
---
paper_title: Multidimensional image value assessment and rating for automated albuming and retrieval

paper_content:

The ability to automatically assess image characteristics is an important function for content management, building digital image albums, storytelling with images, and retrieval of specific visual content. This capability is needed to organize and sort large numbers of image and video assets. This paper proposes a novel approach to assess and rate images based on multidimensional characteristics including image quality, social relationships, aesthetic quality, important events, and usage. This new approach provides additional flexibility for end user applications that utilize different aspects of image characteristics. Specifically, we describe a method for assessing image quality based upon technical characteristics of the image, and for predicting the significance of an image based upon the people portrayed in the image. Experimental results indicate that the proposed multidimensional approach provides a promising framework for automated image value assessment and rating.
---
paper_title: The Design of High-Level Features for Photo Quality Assessment

paper_content:

We propose a principled method for designing high level features forphoto quality assessment. Our resulting system can classify between high quality professional photos and low quality snapshots. Instead of using the bag of low-level features approach, we first determine the perceptual factors that distinguish between professional photos and snapshots. Then, we design high level semantic features to measure the perceptual differences. We test our features on a large and diverse dataset and our system is able to achieve a classification rate of 72% on this difficult task. Since our system is able to achieve a precision of over 90% in low recall scenarios, we show excellent results in a web image search application.
---
paper_title: Towards unsupervised whole-object segmentation: Combining automated matting with boundary detection

paper_content:

We propose a novel step toward the unsupervised segmentation of whole objects by combining ldquohintsrdquo of partial scene segmentation offered by multiple soft, binary mattes. These mattes are implied by a set of hypothesized object boundary fragments in the scene. Rather than trying to find or define a single ldquobestrdquo segmentation, we generate multiple segmentations of an image. This reflects contemporary methods for unsupervised object discovery from groups of images, and it allows us to define intuitive evaluation metrics for our sets of segmentations based on the accurate and parsimonious delineation of scene objects. Our proposed approach builds on recent advances in spectral clustering, image matting, and boundary detection. It is demonstrated qualitatively and quantitatively on a dataset of scenes and is suitable for current work in unsupervised object discovery without top-down knowledge.
---
paper_title: Image Segmentation Evaluation: A Survey of Unsupervised Methods

paper_content:

Image segmentation is an important processing step in many image, video and computer vision applications. Extensive research has been done in creating many different approaches and algorithms for image segmentation, but it is still difficult to assess whether one algorithm produces more accurate segmentations than another, whether it be for a particular image or set of images, or more generally, for a whole class of images. To date, the most common method for evaluating the effectiveness of a segmentation method is subjective evaluation, in which a human visually compares the image segmentation results for separate segmentation algorithms, which is a tedious process and inherently limits the depth of evaluation to a relatively small number of segmentation comparisons over a predetermined set of images. Another common evaluation alternative is supervised evaluation, in which a segmented image is compared against a manually-segmented or pre-processed reference image. Evaluation methods that require user assistance, such as subjective evaluation and supervised evaluation, are infeasible in many vision applications, so unsupervised methods are necessary. Unsupervised evaluation enables the objective comparison of both different segmentation methods and different parameterizations of a single method, without requiring human visual comparisons or comparison with a manually-segmented or pre-processed reference image. Additionally, unsupervised methods generate results for individual images and images whose characteristics may not be known until evaluation time. Unsupervised methods are crucial to real-time segmentation evaluation, and can furthermore enable self-tuning of algorithm parameters based on evaluation results. In this paper, we examine the unsupervised objective evaluation methods that have been proposed in the literature. An extensive evaluation of these methods are presented. The advantages and shortcomings of the underlying design mechanisms in these methods are discussed and analyzed through analytical evaluation and empirical evaluation. Finally, possible future directions for research in unsupervised evaluation are proposed.
---
paper_title: Discovery of social relationships in consumer photo collections using Markov Logic

paper_content:

We identify the social relationships between individuals in consumer photos. Consumer photos generally do not contain a random gathering of strangers but rather groups of friends and families. Detecting and identifying these relationships are important steps towards understanding consumer image collections. Similar to the approach that a human might use, we use a rule-based system to quantify the domain knowledge (e.g. children tend to be photographed more often than adults; parents tend to appear with their kids). The weight of each rule reflects its importance in the overall prediction model. Learning and inference are based on a sound mathematical formulation using the theory developed in the area of statistical relational models. In particular, we use the language called Markov Logic [14]. We evaluate our model using cross validation on a set of about 4500 photos collected from 13 different users. Our experiments show the potential of our approach by improving the accuracy (as well as other statistical measures) over a set of two different relationship prediction tasks when compared with different baselines. We conclude with directions for future work.
---
paper_title: Incorporating camera metadata for attended region detection and consumer photo classification

paper_content:

Photos taken by human beings significantly differ from the pictures that are taken by a surveillance camera or a vision sensor on a robot, e.g., human beings may intentionally capture photos to express his/her feeling or record a memorial scene. Such a creative photo capture process is accomplished by adjusting two factors: (1) the parameters setting of a camera; and (2) the position between the camera and the interesting objects or scenes. To enable automatic understanding and interpretation of the semantics of photos, it is very important to take all these factors into account. Unfortunately, most existing algorithms for image understanding focus on only the content of the images while completely ignoring these two important factors. In this paper, we have developed a new algorithm to calculate what the interestingness of the photographer is and what the core content of a photo is. The gained information (i.e., attended regions and attention of the photographer) is further used to support more effective photo classification and retrieval. Our experiments on 70,000+ photos taken by 200+ different models of cameras have obtained very positive results.
---
paper_title: A computationally efficient approach to indoor/outdoor scene classification

paper_content:

Prior research in scene classification has shown that high-level information can be inferred from low-level image features. Classification rates of roughly 90% have been reported using low-level features to predict indoor scenes vs. outdoor scenes. However, the high classification rates are often achieved by using computationally expensive, high-dimensional feature sets, thus limiting the practical implementation of such systems. We show that a more computationally efficient approach to indoor/outdoor classification can yield classification rates comparable to the best methods reported in the literature. A low complexity, low-dimensional feature set is used in conjunction with a two-stage support vector machine classification scheme to achieve a classification rate of 90.2% on a large database of consumer photographs.
---
paper_title: Global pose estimation using non-tree models

paper_content:

We propose a novel global pose estimation method to detect body parts of articulated objects in images based on non-tree graph models. There are two kinds of edges defined in the body part relation graph: Strong (tree) edges corresponding to the body plan that can enforce any type of constraint, and weak (non-tree) edges that express exclusion constraints arising from inter-part occlusion and symmetry conditions. We express optimal part localization as a multiple shortest path problem in a set of correlated trellises constructed from the graph model. Strong model edges generate the trellises, while weak model edges prohibit implausible poses by generating exclusion constraints among trellis nodes and edges. The optimization may be expressed as an integer linear program and solved using a novel two-stage relaxation scheme. Experiments show that the proposed method has a high chance of obtaining the globally optimal pose at low computational cost.
---
paper_title: Semantic Representation and Recognition of Continued and Recursive Human Activities

paper_content:

This paper describes a methodology for automated recognition of complex human activities. The paper proposes a general framework which reliably recognizes high-level human actions and human-human interactions. Our approach is a description-based approach, which enables a user to encode the structure of a high-level human activity as a formal representation. Recognition of human activities is done by semantically matching constructed representations with actual observations. The methodology uses a context-free grammar (CFG) based representation scheme as a formal syntax for representing composite activities. Our CFG-based representation enables us to define complex human activities based on simpler activities or movements. Our system takes advantage of both statistical recognition techniques from computer vision and knowledge representation concepts from traditional artificial intelligence. In the low-level of the system, image sequences are processed to extract poses and gestures. Based on the recognition of gestures, the high-level of the system hierarchically recognizes composite actions and interactions occurring in a sequence of image frames. The concept of hallucinations and a probabilistic semantic-level recognition algorithm is introduced to cope with imperfect lower-layers. As a result, the system recognizes human activities including `fighting' and `assault', which are high-level activities that previous systems had difficulties. The experimental results show that our system reliably recognizes sequences of complex human activities with a high recognition rate.
---
paper_title: Classification of Digital Photos Taken by Photographers or Home Users

paper_content:

In this paper, we address a specific image classification task, i.e. to group images according to whether they were taken by photographers or home users. Firstly, a set of low-level features explicitly related to such high-level semantic concept are investigated together with a set of general-purpose low-level features. Next, two different schemes are proposed to find out those most discriminative features and feed them to suitable classifiers: one resorts to boosting to perform feature selection and classifier training simultaneously; the other makes use of the information of the label by Principle Component Analysis for feature re-extraction and feature de-correlation; followed by Maximum Marginal Diversity for feature selection and Bayesian classifier or Support Vector Machine for classification. In addition, we show an application in No-Reference holistic quality assessment as a natural extension of such image classification. Experimental results demonstrate the effectiveness of our methods.
---
paper_title: An adaptive unsupervised approach toward pixel clustering and color image segmentation

paper_content:

This paper proposes an adaptive unsupervised scheme that could find diverse applications in pattern recognition as well as in computer vision, particularly in color image segmentation. The algorithm, named Ant Colony-Fuzzy C-means Hybrid Algorithm (AFHA), adaptively clusters image pixels viewed as three dimensional data pieces in the RGB color space. The Ant System (AS) algorithm is applied for intelligent initialization of cluster centroids, which endows clustering with adaptivity. Considering algorithmic efficiency, an ant subsampling step is performed to reduce computational complexity while keeping the clustering performance close to original one. Experimental results have demonstrated AFHA clustering's advantage of smaller distortion and more balanced cluster centroid distribution over FCM with random and uniform initialization. Quantitative comparisons with the X-means algorithm also show that AFHA makes a better pre-segmentation scheme over X-means. We further extend its application to natural image segmentation, taking into account the spatial information and conducting merging steps in the image space. Extensive tests were taken to examine the performance of the proposed scheme. Results indicate that compared with classical segmentation algorithms such as mean shift and normalized cut, our method could generate reasonably good or better image partitioning, which illustrates the method's practical value.
---
paper_title: Familiarity based unified visual attention model for fast and robust object recognition

paper_content:

Even though visual attention models using bottom-up saliency can speed up object recognition by predicting object locations, in the presence of multiple salient objects, saliency alone cannot discern target objects from the clutter in a scene. Using a metric named familiarity, we propose a top-down method for guiding attention towards target objects, in addition to bottom-up saliency. To demonstrate the effectiveness of familiarity, the unified visual attention model (UVAM) which combines top-down familiarity and bottom-up saliency is applied to SIFT based object recognition. The UVAM is tested on 3600 artificially generated images containing COIL-100 objects with varying amounts of clutter, and on 126 images of real scenes. The recognition times are reduced by 2.7x and 2x, respectively, with no reduction in recognition accuracy, demonstrating the effectiveness and robustness of the familiarity based UVAM.
---
paper_title: Shape Based Detection and Top-Down Delineation Using Image Segments

paper_content:

We introduce a segmentation-based detection and top-down figure-ground delineation algorithm. Unlike common methods which use appearance for detection, our method relies primarily on the shape of objects as is reflected by their bottom-up segmentation. ::: ::: Our algorithm receives as input an image, along with its bottom-up hierarchical segmentation. The shape of each segment is then described both by its significant boundary sections and by regional, dense orientation information derived from the segment's shape using the Poisson equation. Our method then examines multiple, overlapping segmentation hypotheses, using their shape and color, in an attempt to find a "coherent whole," i.e., a collection of segments that consistently vote for an object at a single location in the image. Once an object is detected, we propose a novel pixel-level top-down figure-ground segmentation by "competitive coverage" process to accurately delineate the boundaries of the object. In this process, given a particular detection hypothesis, we let the voting segments compete for interpreting (covering) each of the semantic parts of an object. Incorporating competition in the process allows us to resolve ambiguities that arise when two different regions are matched to the same object part and to discard nearby false regions that participated in the voting process. ::: ::: We provide quantitative and qualitative experimental results on challenging datasets. These experiments demonstrate that our method can accurately detect and segment objects with complex shapes, obtaining results comparable to those of existing state of the art methods. Moreover, our method allows us to simultaneously detect multiple instances of class objects in images and to cope with challenging types of occlusions such as occlusions by a bar of varying size or by another object of the same class, that are difficult to handle with other existing class-specific top-down segmentation methods.
---
paper_title: A Regularized Framework for Feature Selection in Face Detection and Authentication

paper_content:

This paper proposes a general framework for selecting features in the computer vision domain-i.e., learning descriptions from data-where the prior knowledge related to the application is confined in the early stages. The main building block is a regularization algorithm based on a penalty term enforcing sparsity. The overall strategy we propose is also effective for training sets of limited size and reaches competitive performances with respect to the state-of-the-art. To show the versatility of the proposed strategy we apply it to both face detection and authentication, implementing two modules of a monitoring system working in real time in our lab. Aside from the choices of the feature dictionary and the training data, which require prior knowledge on the problem, the proposed method is fully automatic. The very good results obtained in different applications speak for the generality and the robustness of the framework.
---
paper_title: Spatial Gaussian Mixture Model for gender recognition

paper_content:

Patch-based approaches have become popular in many computer vision applications over recent years. An intrinsic flaw of this framework, missing of the spatial information, however, restricts its usage in face related applications where the spatial configuration is relatively settled. In this paper, we introduce a new patch feature representation, namely Spatial Gaussian Mixture Models (SGMM), which enhances the traditional GMM approach by taking the spatial information into consideration at both local and global scales. In the meantime, SGMM inherits all the merits of GMM, such as precise appearance description and robustness to image misalignment. The experiments on gender recognition demonstrate that the SGMM representation achieves more than 40% relative error reduction compared with either GMM or SVM-based approaches.
---
paper_title: Prior Knowledge, Level Set Representations & Visual Grouping

paper_content:

Abstract ::: In this paper, we propose a level set method for shape-driven object extraction. We introduce a voxel-wise probabilistic level set formulation to account for prior knowledge. To this end, objects are represented in an implicit form. Constraints on the segmentation process are imposed by seeking a projection to the image plane of the prior model modulo a similarity transformation. The optimization of a statistical metric between the evolving contour and the model leads to motion equations that evolve the contour toward the desired image properties while recovering the pose of the object in the new image. Upon convergence, a solution that is similarity invariant with respect to the model and the corresponding transformation are recovered. Promising experimental results demonstrate the potential of such an approach.
---
paper_title: Aesthetic quality assessment of consumer photos with faces

paper_content:

Automatically assessing the subjective quality of a photo is a challenging area in visual computing. Previous works study the aesthetic quality assessment on a general set of photos regardless of the photo's content and mainly use features extracted from the entire image. In this work, we focus on a specific genre of photos: consumer photos with faces. This group of photos constitutes an important part of consumer photo collections. We first conduct an online study on Mechanical Turk to collect ground-truth and subjective opinions for a database of consumer photos with faces. We then extract technical features, perceptual features, and social relationship features to represent the aesthetic quality of a photo, by focusing on face-related regions. Experiments show that our features perform well for categorizing or predicting the aesthetic quality.
---
paper_title: Consumer image retrieval by estimating relation tree from family photo collections

paper_content:

In this paper, we propose an approach to automatically estimate relationship among people in a family image collection based on results from face analyses technologies including automated face recognition and clustering, demographic assessment, and face similarity measurement, as well as contextual information such as people co-appearance, people's relative positions in photos and image timestamps. As the result, a relation tree can be estimated which provides important semantic information regarding people involved in a photo collection and has numerous applications in photo sharing and browsing, social networking, etc. The methods for deriving and integrating information from photos and the process for estimating a relation tree are described. Experimental results on two typical consumer photo collections and examples of using these results in consumer image retrieval are presented.
---
paper_title: Image Segmentation by MAP-ML Estimations

paper_content:

Image segmentation plays an important role in computer vision and image analysis. In this paper, image segmentation is formulated as a labeling problem under a probability maximization framework. To estimate the label configuration, an iterative optimization scheme is proposed to alternately carry out the maximum a posteriori (MAP) estimation and the maximum likelihood (ML) estimation. The MAP estimation problem is modeled with Markov random fields (MRFs) and a graph cut algorithm is used to find the solution to the MAP estimation. The ML estimation is achieved by computing the means of region features in a Gaussian model. Our algorithm can automatically segment an image into regions with relevant textures or colors without the need to know the number of regions in advance. Its results match image edges very well and are consistent with human perception. Comparing to six state-of-the-art algorithms, extensive experiments have shown that our algorithm performs the best.
---
paper_title: Multivariate Image Segmentation Using Semantic Region Growing With Adaptive Edge Penalty

paper_content:

Multivariate image segmentation is a challenging task, influenced by large intraclass variation that reduces class distinguishability as well as increased feature space sparseness and solution space complexity that impose computational cost and degrade algorithmic robustness. To deal with these problems, a Markov random field (MRF) based multivariate segmentation algorithm called "multivariate iterative region growing using semantics" (MIRGS) is presented. In MIRGS, the impact of intraclass variation and computational cost are reduced using the MRF spatial context model incorporated with adaptive edge penalty and applied to regions. Semantic region growing starting from watershed over-segmentation and performed alternatively with segmentation gradually reduces the solution space size, which improves segmentation effectiveness. As a multivariate iterative algorithm, MIRGS is highly sensitive to initial conditions. To suppress initialization sensitivity, it employs a region-level k -means (RKM) based initialization method, which consistently provides accurate initial conditions at low computational cost. Experiments show the superiority of RKM relative to two commonly used initialization methods. Segmentation tests on a variety of synthetic and natural multivariate images demonstrate that MIRGS consistently outperforms three other published algorithms.
---
paper_title: Clothing cosegmentation for recognizing people

paper_content:

Researches have verified that clothing provides information about the identity of the individual. To extract features from the clothing, the clothing region first must be localized or segmented in the image. At the same time, given multiple images of the same person wearing the same clothing, we expect to improve the effectiveness of clothing segmentation. Therefore, the identity recognition and clothing segmentation problems are inter-twined; a good solution for one aides in the solution for the other. We build on this idea by analyzing the mutual information between pixel locations near the face and the identity of the person to learn a global clothing mask. We segment the clothing region in each image using graph cuts based on a clothing model learned from one or multiple images believed to be the same person wearing the same clothing. We use facial features and clothing features to recognize individuals in other images. The results show that clothing segmentation provides a significant improvement in recognition accuracy for large image collections, and useful clothing masks are simultaneously produced. A further significant contribution is that we introduce a publicly available consumer image collection where each individual is identified. We hope this dataset allows the vision community to more easily compare results for tasks related to recognizing people in consumer image collections.
---
paper_title: Learning to Combine Bottom-Up and Top-Down Segmentation

paper_content:

Bottom-up segmentation based only on low-level cues is a notoriously difficult problem. This difficulty has lead to recent top-down segmentation algorithms that are based on class-specific image information. Despite the success of top-down algorithms, they often give coarse segmentations that can be significantly refined using low-level cues. This raises the question of how to combine both top-down and bottom-up cues in a principled manner. ::: ::: In this paper we approach this problem using supervised learning. Given a training set of ground truth segmentations we train a fragment-based segmentation algorithm which takes into account both bottom-up and top-down cues simultaneously, in contrast to most existing algorithms which train top-down and bottom-up modules separately. We formulate the problem in the framework of Conditional Random Fields (CRF) and derive a feature induction algorithm for CRF, which allows us to efficiently search over thousands of candidate fragments. Whereas pure top-down algorithms often require hundreds of fragments, our simultaneous learning procedure yields algorithms with a handful of fragments that are combined with low-level cues to efficiently compute high quality segmentations.
---
paper_title: Discriminative local binary patterns for human detection in personal album

paper_content:

In recent years, local pattern based object detection and recognition have attracted increasing interest in computer vision research community. However, to our best knowledge no previous work has focused on utilizing local patterns for the task of human detection. In this paper we develop a novel human detection system in personal albums based on LBP (local binary pattern) descriptor. Firstly we review the existing gradient based local features widely used in human detection, analyze their limitations and argue that LBP is more discriminative. Secondly, original LBP descriptor does not suit the human detecting problem well due to its high complexity and lack of semantic consistency, thus we propose two variants of LBP: Semantic-LBP and Fourier-LBP. Carefully designed experiments demonstrate the superiority of LBP over other traditional features for human detection. Especially we adopt a random ensemble algorithm for better comparison between different descriptors. All experiments are conducted on INRIA human database.
---
paper_title: Contextual Identity Recognition in Personal Photo Albums

paper_content:

We present an efficient probabilistic method for identity recognition in personal photo albums. Personal photos are usually taken under uncontrolled conditions -the captured faces exhibit significant variations in pose, expression and illumination that limit the success of traditional face recognition algorithms. We show how to improve recognition rates by incorporating additional cues present in personal photo collections, such as clothing appearance and information about when the photo was taken. This is done by constructing a Markov random field (MRF) that effectively combines all available contextual cues in a principled recognition framework. Performing inference in the MRF produces markedly improved recognition results in a challenging dataset consisting of the personal photo collections of multiple people. At the same time, the computational cost of our approach remains comparable to that of standard face recognition approaches.
---
paper_title: Context and observation driven latent variable model for human pose estimation

paper_content:

Current approaches to pose estimation and tracking can be classified into two categories: generative and discriminative. While generative approaches can accurately determine human pose from image observations, they are computationally expensive due to search in the high dimensional human pose space. On the other hand, discriminative approaches do not generalize well, but are computationally efficient. We present a hybrid model that combines the strengths of the two in an integrated learning and inference framework. We extend the Gaussian process latent variable model (GPLVM) to include an embedding from observation space (the space of image features) to the latent space. GPLVM is a generative model, but the inclusion of this mapping provides a discriminative component, making the model observation driven. Observation Driven GPLVM (OD-GPLVM) not only provides a faster inference approach, but also more accurate estimates (compared to GPLVM) in cases where dynamics are not sufficient for the initialization of search in the latent space. We also extend OD-GPLVM to learn and estimate poses from parameterized actions/gestures. Parameterized gestures are actions which exhibit large systematic variation in joint angle space for different instances due to difference in contextual variables. For example, the joint angles in a forehand tennis shot are function of the height of the ball (Figure 2). We learn these systematic variations as a function of the contextual variables. We then present an approach to use information from scene/objects to provide context for human pose estimation for such parameterized actions.
---
paper_title: Generalizing Edge Detection to Contour Detection for Image Segmentation

paper_content:

One approach to image segmentation defines a function of image partitions whose maxima correspond to perceptually salient segments. We extend previous approaches following this framework by requiring that our image model sharply decreases in its power to organize the image as a segment's boundary is perturbed from its true position. Instead of making segment boundaries prefer image edges, we add a term to the objective function that seeks a sharp change in fitness with respect to the entire contour's position, generalizing from edge detection's search for sharp changes in local image brightness. We also introduce a prior on the shape of a salient contour that expresses the observed multi-scale distribution of contour curvature for physical contours. We show that our new term correlates strongly with salient structure. We apply our method to real images and verify that the new term improves performance. Comparisons with other state-of-the-art approaches validate our method's advantages.
---
paper_title: Performance evaluation of local features in human classification and detection

paper_content:

Detecting pedestrians accurately is the first fundamental step for many computer vision applications such as video surveillance, smart vehicles, intersection traffic analysis and so on. The authors present an experimental study on pedestrian detection using state-of-the-art local feature extraction and support vector machine (SVM) classifiers. The performance of pedestrian detection using region covariance, histogram of oriented gradients (HOG) and local receptive fields (LRF) feature descriptors is experimentally evaluated. The experiments are performed on the DaimlerChrysler benchmarking data set, the MIT CBCL data set and 'Intitut National de Recherche en Informatique et Automatique (INRIA) data set. All can be publicly accessed. The experimental results show that region covariance features with radial basis function kernel SVM and HOG features with quadratic kernel SVM outperform the combination of LRF features with quadratic kernel SVM. Furthermore, the results reveal that both covariance and HOG features perform very well in the context of pedestrian detection.
---
paper_title: Blur detection for digital images using wavelet transform

paper_content:

With the prevalence of digital cameras, the number of digital images increases quickly, which raises the demand for image quality assessment in terms of blur. Based on the edge type and sharpness analysis, using the Harr wavelet transform, a new blur detection scheme is proposed in this paper, which can determine whether an image is blurred or not and to what extent an image is blurred. Experimental results demonstrate the effectiveness of the proposed scheme.
---
paper_title: Unsupervised colour image segmentation using dual-tree complex wavelet transform

paper_content:

In this paper we present an effective unsupervised colour image segmentation algorithm which uses multiscale edge information and spatial colour content. The multiscale edge information is extracted using the dual-tree complex wavelet transform. Binary morphological operators are applied to the edge information to detect seed regions which are large enough to exclude boundary-only regions. The segmentation of homogeneous regions is obtained using region growing followed by region merging in the CIE L^*a^*b^* colour space. We also present an edge preserving smoothing filter as a pre-process for the algorithm. We compare our algorithm with state-of-the-art algorithms and show its superior performance.
---
paper_title: Normalized tree partitioning for image segmentation

paper_content:

In this paper, we propose a novel graph based clustering approach with satisfactory clustering performance and low computational cost. It consists of two main steps: tree fitting and partitioning. We first introduce a probabilistic method to fit a tree to a data graph under the sense of minimum entropy. Then, we propose a novel tree partitioning method under a normalized cut criterion, called normalized tree partitioning (NTP), in which a fast combinatorial algorithm is designed for exact bipartitioning. Moreover, we extend it to k-way tree partitioning by proposing an efficient best-first recursive bipartitioning scheme. Compared with spectral clustering, NTP produces the exact global optimal bipartition, introduces fewer approximations for k-way partitioning and can intrinsically produce superior performance. Compared with bottom-up aggregation methods, NTP adopts a global criterion and hence performs better. Last, experimental results on image segmentation demonstrate that our approach is more powerful compared with existing graph-based approaches.
---
paper_title: Text detection in personal image collections

paper_content:

We present a novel text detection technique for the application of image annotation in databases of consumer pictures and videos. Images are annotated using information only from the picture itself, as a regular consumer would do. We present results using the challenging public ICDAR2003 dataset and show that our algorithm is competitive with respect to state-of-the-art.
---
paper_title: Measuring social networks with digital photograph collections

paper_content:

The ease and lack of cost associated with taking digital photographs have allowed people to amass large personal photograph collections. These collections contain valuable information about their owners' social relationships. This paper is a preliminary investigation into how digital photo collections can provide useful data for the study of social networks. Results from an analysis of 23 subjects' photo collections demonstrate the feasibility of this approach. The relationship between perceived closeness and network position, as well as future questions, are also discussed.
---
paper_title: Leveraging personal photos to inferring friendships in social network services

paper_content:

Social network services have become widely used as an important tool to share rich information; in such networks, making new friends is the most basic functionality to enable users to take advantage of their social networks. In this paper we look into personal photos as an additional source for social network analysis and analyze the potential of people tags in the photos for friend recommendations. We also propose a new compact data structure, collectively called Face Co-Occurrence Networks (FCON), which stores crucial and quantitative information about people's appearance in photos. We discover strong associative relationships among people and recommend reliable social friends by utilizing FCON. Experimental results demonstrate the effectiveness and efficiency of our method for recommending friends in social network services.
---
paper_title: Automatic skin segmentation and tracking in sign language recognition

paper_content:

Skin segmentation and tracking play an important role in sign language recognition. A framework for segmenting and tracking skin objects from signing videos is described. It mainly consists of two parts: a skin colour model and a skin object tracking system. The skin colour model is first built based on the combination of support vector machine active learning and region segmentation. Then, the obtained skin colour model is integrated with the motion and position information to perform segmentation and tracking. The tracking system is able to predict occlusions among any of the skin objects using a Kalman filter (KF). Moreover, the skin colour model can be updated with the help of tracking to handle illumination variation. Experimental evaluations using real-world gesture videos and comparison with other existing algorithms demonstrate the effectiveness of the proposed work.
---
paper_title: Fast Human Pose Estimation using Appearance and Motion via Multi-Dimensional Boosting Regression

paper_content:

We address the problem of estimating human pose in video sequences, where rough location has been determined. We exploit both appearance and motion information by defining suitable features of an image and its temporal neighbors, and learning a regression map to the parameters of a model of the human body using boosting techniques. Our algorithm can be viewed as a fast initialization step for human body trackers, or as a tracker itself. We extend gradient boosting techniques to learn a multi-dimensional map from (rotated and scaled) Haar features to the entire set of joint angles representing the full body pose. We test our approach by learning a map from image patches to body joint angles from synchronized video and motion capture walking data. We show how our technique enables learning an efficient real-time pose estimator, validated on publicly available datasets.
---
paper_title: The role of image composition in image aesthetics

paper_content:

In this paper, we explore the role that image composition plays in image aesthetic appeal classification. We propose low-level image composition features that approximate traditional photography composition guidelines, such as simplicity and visual balance (e.g. golden mean, golden triangles). We then use these features to build an image aesthetics classifier that we test with a baseline dataset. Interestingly, our approach, that only takes into account image composition features, yields close to state-of-the-art image aesthetic-based classification accuracies, which highlights the importance of image composition in image aesthetic appeal assessment.
---
paper_title: Annotating collections of photos using hierarchical event and scene models

paper_content:

Most image annotation systems consider a single photo at a time and label photos individually. In this work, we focus on collections of personal photos and explore the associated GPS and time information for semantic annotation. First, we employ a constrained clustering method to partition a photo collection into event-based sub-collections, considering that the GPS records may be partly missing (a practical issue). We then use conditional random field (CRF) models to exploit the correlation between photos based on (1) time-location constraints and (2) the relationship between collection-level annotation (i.e., events) and image-level annotation (i.e., scenes). With the introduction of such a multi-level annotation hierarchy, our system addresses the problem of annotating consumer photo collections that requires a more hierarchical description of the customerspsila activities than do the simpler image annotation tasks. The efficacy of the proposed system is validated using a geotagged customer photo collection database, which consists of over 100 folders and is labeled for 12 events and 12 scenes.
---
paper_title: Resolution-Invariant Image Representation for Content-Based Zooming

paper_content:

This paper presents a novel Resolution-Invariant Image Representation (ЯIIR) framework, and applies it for Content-Based Zooming (CBZ) applications. We explain how to generate a multi-resolution bases set, from which the learned image representation can be resolution-invariant. This provides the key technology to support the continues image up-scaling task for the CBZ applications, which existing example-based resolution enhancement approaches cannot handel, or simply 2-D image interpolation algorithm cannot give satisfactory image quality for. We discuss two clustering based methods to construct the bases set. Experimental results show that, both the two methods give good image quality, and the proposed ЯIIR framework outperforms existing methods in various aspects.
---
paper_title: Local probabilistic descriptors for image categorisation

paper_content:

Image categorisation involves the well known difficulties with different visual appearances of a single object, but also introduces the problem of within-category variation. This within-category variation makes highly distinctive local descriptors less appropriate for categorisation. Difficulties because of the within category variation and clutter are tackled by modelling image fragments in a new manner. The authors propose a family of local image descriptors, called probabilistic patch descriptors (PPDs). PPDs encode the appearance of image fragments as well as their variability within a category. PPDs extend the usual local descriptors by also modelling the variance of the descriptors' elements, for example pixels or bins in a histogram. To compare two PPDs, and a PPD with an image, a new similarity measure called PPD matching score is introduced. For each object category, a set of representative PPDs is learnt. Images are represented as feature vectors of the best matching scores obtained for representative PPDs in images. Support vector machine classifiers are then trained on the feature vectors. PPDs are applied to image categorisation using machine learning where the features are the matching scores between images and PPDs. The authors experiment with two variants of PPDs that are based on complementary local descriptors. An interesting observation is that combining the two PPD variants improves the accuracy of categorisation. Experiments indicate that the benefits of modelling the within-category variation give results that are comparable with the state-of-the-art categorisation methods, and show good robustness with respect to noise and occlusions.
---
paper_title: Temporal event clustering for digital photo collections

paper_content:

We present similarity-based methods to cluster digital photos by time and image content. The approach is general, unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present results for the algorithm based solely on temporal similarity, and jointly on temporal and content-based similarity. We also describe a supervised algorithm based on learning vector quantization. Finally, we include experimental results for the proposed algorithms and several competing approaches on two test collections.
---
paper_title: Joint multi-label multi-instance learning for image classification

paper_content:

In real world, an image is usually associated with multiple labels which are characterized by different regions in the image. Thus image classification is naturally posed as both a multi-label learning and multi-instance learning problem. Different from existing research which has considered these two problems separately, we propose an integrated multi- label multi-instance learning (MLMIL) approach based on hidden conditional random fields (HCRFs), which simultaneously captures both the connections between semantic labels and regions, and the correlations among the labels in a single formulation. We apply this MLMIL framework to image classification and report superior performance compared to key existing approaches over the MSR Cambridge (MSRC) and Corel data sets.
---
paper_title: Event classification in personal image collections

paper_content:

In this paper, we investigate event classification that is specifically developed for use in consumer family photo collections. This domain is very different from news video collections that have been the focus of research in the area of scene content classification. We determine a set of broad event classes that are relevant to personal collections. We investigate the use of a variety of high-level visual and temporal features, and determine a set of features that show good correlation with the event class. We propose a Bayesian belief network for event classification that computes the a posteriori probability of the event class given the input features. The Bayes net is trained on a large set of manually annotated consumer collections. We obtain a classification accuracy of over 70% in this challenging domain.
---
paper_title: Categorization of natural scenes: Local versus global information and the role of color

paper_content:

Categorization of scenes is a fundamental process of human vision that allows us to efficiently and rapidly analyze our surroundings. Several studies have explored the processes underlying human scene categorization, but they have focused on processing global image information. In this study, we present both psychophysical and computational experiments that investigate the role of local versus global image information in scene categorization. In a first set of human experiments, categorization performance is tested when only local or only global image information is present. Our results suggest that humans rely on local, region-based information as much as on global, configural information. In addition, humans seem to integrate both types of information for intact scene categorization. In a set of computational experiments, human performance is compared to two state-of-the-art computer vision approaches that have been shown to be psychophysically plausible and that model either local or global information. In addition to the influence of local versus global information, in a second series of experiments, we investigated the effect of color on the categorization performance of both the human observers and the computational model. Analysis of the human data suggests that color is an additional channel of perceptual information that leads to higher categorization results at the expense of increased reaction times in the intact condition. However, it does not affect reaction times when only local information is present. When color is removed, the employed computational model follows the relative performance decrease of human observers for each scene category and can thus be seen as a perceptually plausible model for human scene categorization based on local image information.
---
paper_title: PORNOGRAPHIC IMAGE DETECTION BASED ON MULTILEVEL REPRESENTATION

paper_content:

With the proliferation of pornographic images on the Internet, it is essential to automatically detect pornographic images by analyzing image content. Most traditional detection systems are based on low-level features and generate many false positives due to images that contain large regions of skin-like colors. In this paper, we present a novel detection method based on local features, such as SIFT (Scale Invariant Feature Transform) visual words. Support Vector Machine (SVM) is used to classify images according to their multilevel representation based on visual words and the distribution of pornography-related visual words. The multilevel representation captures inter-word statistics and fuses various visual components of pornographic scenes. Experimental results demonstrate that our method outperforms traditional skin-region and human-body-model based methods, and performs well on a wide range of test data, in particular, on human-related images.
---
paper_title: Content-Based Photo Quality Assessment

paper_content:

Automatically assessing photo quality from the perspective of visual aesthetics is of great interest in high-level vision research and has drawn much attention in recent years. In this paper, we propose content-based photo quality assessment using regional and global features. Under this framework, subject areas, which draw the most attentions of human eyes, are first extracted. Then regional features extracted from subject areas and the background regions are combined with global features to assess the photo quality. Since professional photographers may adopt different photographic techniques and may have different aesthetic criteria in mind when taking different types of photos (e.g. landscape versus portrait), we propose to segment regions and extract visual features in different ways according to the categorization of photo content. Therefore we divide the photos into seven categories based on their content and develop a set of new subject area extraction methods and new visual features, which are specially designed for different categories. This argument is supported by extensive experimental comparisons of existing photo quality assessment approaches as well as our new regional and global features over different categories of photos. Our new features significantly outperform the state-of-the-art methods. Another contribution of this work is to construct a large and diversified benchmark database for the research of photo quality assessment. It includes 17, 613 photos with manually labeled ground truth.
---
paper_title: Classifying Images Collected on the World Wide Web

paper_content:

This work presents the classification of images collected on the World Wide Web, using a supervised classification method, called ID3 (Itemized Dichotomizer 3). The classification consists in separating the images into two semantic classes: graphics and photographs. Photographs include natural scenes, like people, faces, animals, flowers, landscapes and cities. Graphics are logos, drawings, icons, maps, and backgrounds, usually generated by computer. To validate the classifier we used the k-fold cross-validation method. In the experimental tests 95.6% of the images were correctly classified.
---
paper_title: Two-Dimensional Active Learning for image classification

paper_content:

In this paper, we propose a two-dimensional active learning scheme and show its application in image classification. Traditional active learning methods select samples only along the sample dimension. While this is the right strategy in binary classification, it is sub-optimal for multi-label classification. In multi-label classification, we argue that, for each selected sample, only a part of more effective labels are necessary to be annotated while others can be inferred by exploring the correlations among the labels. The reason is that the contributions of different labels to minimizing the classification error are different due to the inherent label correlations. To this end, we propose to select sample-label pairs, rather than only samples, to minimize a multi-label Bayesian classification error bound. This new active learning strategy not only considers the sample dimension but also the label dimension, and we call it Two-Dimensional Active Learning (2DAL). We also show that the traditional active learning formulation is a special case of 2DAL when there is only one label. Extensive experiments conducted on two real-world applications show that the 2DAL significantly outperforms the best existing approaches which did not take label correlation into account.
---
paper_title: A computationally efficient approach to indoor/outdoor scene classification

paper_content:

Prior research in scene classification has shown that high-level information can be inferred from low-level image features. Classification rates of roughly 90% have been reported using low-level features to predict indoor scenes vs. outdoor scenes. However, the high classification rates are often achieved by using computationally expensive, high-dimensional feature sets, thus limiting the practical implementation of such systems. We show that a more computationally efficient approach to indoor/outdoor classification can yield classification rates comparable to the best methods reported in the literature. A low complexity, low-dimensional feature set is used in conjunction with a two-stage support vector machine classification scheme to achieve a classification rate of 90.2% on a large database of consumer photographs.
---
paper_title: Image classification based on pyramid histogram of topics

paper_content:

In this paper we propose PHOTO (Pyramid Histogram Of TOpics), a new representation for image classification. We partition the image into hierarchical cells and learn the topic histogram using pLSA over each cell with EM algorithm. Then we concatenate the topic histograms over the cells at all levels to form a “long” vector, i.e. pyramid histogram of topics. Finally AdaBoost classifiers are used to select the topics most discriminative for class recognition. Experimental results on two diverse databases show that our method performs significantly better than general topic representation.
---
paper_title: Image Clustering Using Local Discriminant Models and Global Integration

paper_content:

In this paper, we propose a new image clustering algorithm, referred to as clustering using local discriminant models and global integration (LDMGI). To deal with the data points sampled from a nonlinear manifold, for each data point, we construct a local clique comprising this data point and its neighboring data points. Inspired by the Fisher criterion, we use a local discriminant model for each local clique to evaluate the clustering performance of samples within the local clique. To obtain the clustering result, we further propose a unified objective function to globally integrate the local models of all the local cliques. With the unified objective function, spectral relaxation and spectral rotation are used to obtain the binary cluster indicator matrix for all the samples. We show that LDMGI shares a similar objective function with the spectral clustering (SC) algorithms, e.g., normalized cut (NCut). In contrast to NCut in which the Laplacian matrix is directly calculated based upon a Gaussian function, a new Laplacian matrix is learnt in LDMGI by exploiting both manifold structure and local discriminant information. We also prove that K-means and discriminative K-means (DisKmeans) are both special cases of LDMGI. Extensive experiments on several benchmark image datasets demonstrate the effectiveness of LDMGI. We observe in the experiments that LDMGI is more robust to algorithmic parameter, when compared with NCut. Thus, LDMGI is more appealing for the real image clustering applications in which the ground truth is generally not available for tuning algorithmic parameters.
---
paper_title: Aesthetic quality assessment of consumer photos with faces

paper_content:

Automatically assessing the subjective quality of a photo is a challenging area in visual computing. Previous works study the aesthetic quality assessment on a general set of photos regardless of the photo's content and mainly use features extracted from the entire image. In this work, we focus on a specific genre of photos: consumer photos with faces. This group of photos constitutes an important part of consumer photo collections. We first conduct an online study on Mechanical Turk to collect ground-truth and subjective opinions for a database of consumer photos with faces. We then extract technical features, perceptual features, and social relationship features to represent the aesthetic quality of a photo, by focusing on face-related regions. Experiments show that our features perform well for categorizing or predicting the aesthetic quality.
---
paper_title: Embedding spatial information into image content description for scene retrieval

paper_content:

This article presents @D-TSR, an image content representation describing the spatial layout with triangular relationships of visual entities, which can be symbolic objects or low-level visual features. A semi-local implementation of @D-TSR is also proposed, making the description robust to viewpoint changes. We evaluate @D-TSR for image retrieval under the query-by-example paradigm, on contents represented with interest points in a bag-of-features model: it improves state-of-the-art techniques, in terms of retrieval quality as well as of execution time, and is scalable. Finally, its effectiveness is evaluated on a topical scenario dedicated to scene retrieval in datasets of city landmarks.
---
paper_title: Blur detection for digital images using wavelet transform

paper_content:

With the prevalence of digital cameras, the number of digital images increases quickly, which raises the demand for image quality assessment in terms of blur. Based on the edge type and sharpness analysis, using the Harr wavelet transform, a new blur detection scheme is proposed in this paper, which can determine whether an image is blurred or not and to what extent an image is blurred. Experimental results demonstrate the effectiveness of the proposed scheme.
---
paper_title: Summarization of Archived and Shared Personal Photo Collections

paper_content:

The volume of personal photos hosted on photo archives and social sharing platforms has been increasing exponentially. It is difficult to get an overview of a large collection of personal photos without browsing though the entire database manually. In this research, we propose a framework to generate representative subset summaries from photo collections hosted on web archives or social networks. We define salient properties of an effective photo summary and model summarization as an optimization of these properties, given the size constraints. We also introduce metrics for evaluating photo summaries based on their information content and the ability to satisfy user's information needs. Our experiments show that our summarization framework performs better than baseline algorithms.
---
paper_title: Modeling the World from Internet Photo Collections

paper_content:

There are billions of photographs on the Internet, comprising the largest and most diverse photo collection ever assembled. How can computer vision researchers exploit this imagery? This paper explores this question from the standpoint of 3D scene modeling and visualization. We present structure-from-motion and image-based rendering algorithms that operate on hundreds of images downloaded as a result of keyword-based image search queries like "Notre Dame" or "Trevi Fountain." This approach, which we call Photo Tourism, has enabled reconstructions of numerous well-known world sites. This paper presents these algorithms and results as a first step towards 3D modeling of the world's well-photographed sites, cities, and landscapes from Internet imagery, and discusses key open problems and challenges for the research community.
---
paper_title: Creating animation with personal photo collections and map for storytelling

paper_content:

The goal of our research is to support the sharing of stories with digital photographs. Some map sites are now collecting stories associated with peoples' relationships to places. Users are mapping collections of places including intangible emotional associations of places along with photographs, videos. Though this framework of mapping stories is important for accelerating individual creation and transmission of map content, it is not expressive enough to communicate stories narratively. For example, especially when the number of the mapped collections of places is large, it is not easy for viewers to read the map, and it is not easy for creators to express stories as series of events in the real world. That is because one narrative story in a form of a sequence of text narrations, a sequence of photographs, a movie, and an audio like podcasting, etc. is mapped to just one point. As a result, it's up to the viewers which point on the map to read, and in what order. The common framework is rather suitable for mapping and expressing fragments or snapshots of a whole story, and not suitable for expressing a whole story narratively by using a whole area of a map as the setting of the story. We therefore propose a new framework for mapping personal photo collections and constructing them as stories such as route guidances, sightseeing guidances, historical topics, fieldwork records, personal diaries, and so on. We named this framework as spatial slideshow. A spatial slideshow is a fusion of personal photo mapping and photo storytelling. Each story of a spatial slideshow is made of a sequence of mapped photographs, and presented as synchronized animations of a map and an enhanced photo slideshow. The main technical suggestion of this paper is a method for creating a three-dimensional animation of photo image which has a visual effect of moving from photo to photo. We have developed a personal photo album software and released it on the Web, which works as an editor and a browser of spatial slideshows. We assume the proposed framework has a significance of helping with grassroots development of spatial content driven by visual communications about places or events in the real world.
---
paper_title: Detecting and matching repeated patterns for automatic geo-tagging in urban environments

paper_content:

We present a novel method for automatically geo-tagging photographs of man-made environments via detection and matching of repeated patterns. Highly repetitive environments introduce numerous correspondence ambiguities and are problematic for traditional wide-baseline matching methods. Our method exploits the highly repetitive nature of urban environments, detecting multiple perspectively distorted periodic 2D patterns in an image and matching them to a 3D database of textured facades by reasoning about the underlying canonical forms of each pattern. Multiple 2D-to-3D pattern correspondences enable robust recovery of camera orientation and location. We demonstrate the success of this method in a large urban environment.
---
paper_title: Image integrity and aesthetics: towards a more encompassing definition of visual quality

paper_content:

Visual quality is a multifaceted quantity that depends on multiple attributes of the image/video. According to Keelan's ::: definition, artifactual attributes concern features of the image that when visible, are annoying and compromise the ::: integrity of the image. Aesthetic attributes instead depend on the observer's personal taste. Both types of attributes have ::: been studied in the literature in relation to visual quality, but never in conjunction with each other. In this paper we ::: perform a psychometric experiment to investigate how artifactual and aesthetic attributes interact, and how they affect ::: the viewing behavior. In particular, we studied to what extent the appearance of artifacts impacts the aesthetic quality of ::: images. Our results indicate that indeed image integrity somehow influences the aesthetic quality scores. By means of an ::: eye-tracker, we also recorded and analyzed the viewing behavior of our participants while scoring aesthetic quality. ::: Results reveal that, when scoring aesthetic quality, viewing behavior significantly departs from the natural free looking, ::: as well as from the viewing behavior observed for integrity scoring.
---
paper_title: Supporting personal photo storytelling for social albums

paper_content:

Information overload is one of today's major concerns. As high-resolution digital cameras become increasingly pervasive, unprecedented amounts of social media are being uploaded to online social networks on a daily basis. In order to support users on selecting the best photos to create an online photo album, attention has been devoted to the development of automatic approaches for photo storytelling. In this paper, we present a novel photo collection summarization system that learns some of the users' social context by analyzing their online photo albums, and includes storytelling principles and face and image aesthetic ranking in order to assist users in creating new photo albums to be shared online. In an in-depth user study conducted with 12 subjects, the proposed system was validated as a first step in the photo album creation process, helping users reduce workload to accomplish such a task. Our findings suggest that a human audio/video professional with cinematographic skills does not perform better than our proposed system.
---
paper_title: Consumer photo management and browsing facilitated by near-duplicate detection with feature filtering

paper_content:

Near-duplicate detection techniques are exploited to facilitate representative photo selection and region-of-interest (ROI) determination, which are important functionalities for efficient photo management and browsing. To make near-duplicate detection module resist to noisy features, three filtering approaches, i.e., point-based, region-based, and probabilistic latent semantic (pLSA), are developed to categorize feature points. For the photos taken in travels, we construct a support vector machine classifier to model matching patterns between photos and determine whether photos are near-duplicate pairs. Relationships between photos are then described as a graph, and the most central photo that best represents a photo cluster is selected according to centrality values. Because matched feature points are often located in the interior or at the contour of important objects, the region that compactly covers the matched feature points is determined as the ROI. We compare the proposed approaches with conventional ones and demonstrate their effectiveness.
---
paper_title: Personal Photo Organizer Based on Automated Annotation Framework

paper_content:

Current researches toward solving personal photo management suffered two problems: (1) lacking of training data, and (2) no consolidated reference for classification. In this paper, we propose an automated annotation framework to address these problems. The framework was composed by three main components: the context information generator, the semantic concept detector, and the face recognition model. By assigning multi-labels for each photo, the framework makes the photo collection more structured and searchable. Our experimental results show that the techniques used in this framework are promising.
---
paper_title: Indexing personal image collections: a flexible, scalable solution

paper_content:

The growth of personal image collections has boosted the creation of many applications, many of which depend on the existence of fast schemes to match similar image descriptors. In this paper we present multicurves, a new indexing method for multimedia descriptors, able to handle high dimensionalities (100 dimensions and over) and large databases (millions of descriptors). The technique allows a fast implementation of approximate kNN search, and deals easily with data updating (insertions and deletions). The index is based on the simultaneous use of several moderate-dimensional space-filling curves. The combined effect of having more than one curve, and reducing the dimensionality of each individual curve allows overcoming undesirable boundary effects. In empirical evaluations, the method compares favorably with state-of-the-art methods, especially when the constraints of secondary storage are considered.
---
paper_title: Summarizing tourist destinations by mining user-generated travelogues and photos

paper_content:

Automatically summarizing tourist destinations with both textual and visual descriptions is highly desired for online services such as travel planning, to facilitate users to understand the local characteristics of tourist destinations. Travelers are contributing a great deal of user-generated travelogues and photos on the Web, which contain abundant travel-related information and cover various aspects (e.g., landmarks, styles, activities) of most locations in the world. To leverage the collective knowledge of travelers for destination summarization, in this paper we propose a framework which discovers location-representative tags from travelogues and then select relevant and representative photos to visualize these tags. The learnt tags and selected photos are finally organized appropriately to provide an informative summary which describes a given destination both textually and visually. Experimental results based on a large collection of travelogues and photos show promising results on destination summarization.
---
paper_title: Fast near duplicate detection for personal image collections

paper_content:

Due to the rapid growth in personal image collections, there is increasing interest on automatic detection of near duplicates. In this paper, we propose a novel fast near duplicate detection framework that takes advantages of heterogeneous features like EXIF data, global image histogram and local features. To improve the accuracy of local feature matching, we have developed a structure matching algorithm that takes into account of a local feature's neighborhood which can effectively reject mismatches. In addition, we developed a computation-sensitive cascade framework to combine stage classifiers trained on different feature spaces with different computational cost. This method can quickly accept easily identified duplicates using only cheap features without the need to extract more sophisticate but expensive ones. Compared with existing approaches, our experiments show very promising results using our new approach in terms of both efficiency and effectiveness.
---
paper_title: Context-Aware Person Identification in Personal Photo Collections

paper_content:

Identifying the people in photos is an important need for users of photo management systems. We present MediAssist, one such system which facilitates browsing, searching and semi-automatic annotation of personal photos, using analysis of both image content and the context in which the photo is captured. This semi-automatic annotation includes annotation of the identity of people in photos. In this paper, we focus on such person annotation, and propose person identification techniques based on a combination of context and content. We propose language modelling and nearest neighbor approaches to context-based person identification, in addition to novel face color and image color content-based features (used alongside face recognition and body patch features). We conduct a comprehensive empirical study of these techniques using the real private photo collections of a number of users, and show that combining context- and content-based analysis improves performance over content or context alone.
---
paper_title: Signature-image-based event analysis for personal photo albums

paper_content:

Quick reorganizing and draft annotating personal photo albums under event scheme is an emerging trend. In this research, a method has been developed to meet such requirements using the idea of gist and mosaic art so that viewers could understand the meaning of a whole scene without paying much attention in individual details. First, given a photo album, all chronologically ordered images are normalized to a smaller size, and then mosaicked side-by-side to create a signature image representing for that album. Next, by integrating the optimized linear programming with the color descriptor of the signature image, not only the event-type of the album but also all sub-event-types of the sub-sequence photos are decided. More than 19,000 images of five varied event-types have been used to evaluate the proposed method. Experimental results show that the proposed method could detect events towards annotation and re-organization of personal photo albums with high accuracy at a rapid speed.
---
paper_title: Taking the Tags with You: Digital Photograph Provenance

paper_content:

Prior to digital photography, the back of the glossy photo paper served as the canvas to write the important photo details. The photos were then transferred to a series of photo albums. The general public creates too many pictures to sustain the physical photo album. One byproduct of taking digital photos is the automatic number scheme embedded in the digital device. The semantic context of any photo can not be represented with this automatic labeling method. Hence, the user is responsible for concisely renaming the digital images. The rise of digital pictures creates a need for more reliant on storage technology for longevity. We present a suite of annotation tags that capture provenance details. We classify consumer photography provenance in terms of photo manipulation, photo clustering and photo semantics.
---
paper_title: Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition

paper_content:

Managing a large number of digital photos is a challenging task for casual users. Personal photos often don't have rich metadata, or additional information associated with them. However, available metadata can play a crucial role in managing photos. Labeling the semantic content of photos (i.e., annotating them), can increase the amount of metadata and facilitate efficient management. However, manual annotation is tedious and labor intensive while automatic metadata extraction techniques often generate inaccurate and irrelevant results. This paper describes a semi-automatic annotation strategy that takes advantage of human and computer strengths. The semi-automatic approach enables users to efficiently update automatically obtained metadata interactively and incrementally. Even though automatically identified metadata are compromised with inaccurate recognition errors, the process of correcting inaccurate information can be faster and easier than manually adding new metadata from scratch. In this paper, we introduce two photo clustering algorithms for generating meaningful photo groups: (1) Hierarchical event clustering; and (2) Clothing based person recognition, which assumes that people who wear similar clothing and appear in photos taken in one day are very likely to be the same person. To explore our semi-automatic strategies, we designed and implemented a prototype called SAPHARI (Semi-Automatic PHoto Annotation and Recognition Interface). The prototype provides an annotation framework which focuses on making bulk annotations on automatically identified photo groups. The prototype automatically creates photo clusters based on events, people, and file metadata so that users can easily bulk annotation photos. We performed a series of user studies to investigate the effectiveness and usability of the semi-automatic annotation techniques when applied to personal photo collections. The results show that users were able to make annotations significantly faster with event clustering using SAPHARI. We also found that users clearly preferred the semi-automatic approaches.
---
paper_title: An on-line learning method for face association in personal photo collection ☆

paper_content:

Due to the widespread use of cameras, it is very common to collect thousands of personal photos. A proper organization is needed to make the collection usable and to enable an easy photo retrieval. In this paper, we present a method to organize personal photo collections based on ''who'' is in the picture. Our method consists in detecting the faces in the photo sequence and arranging them in groups corresponding to the probable identities. This problem can be conveniently modeled as a multi-target visual tracking where a set of on-line trained classifiers is used to represent the identity models. In contrast to other works where clustering methods are used, our method relies on a probabilistic framework; it does not require any prior information about the number of different identities in the photo album. To enable future comparison, we present experimental results on a public dataset and on a photo collection generated from a public face dataset.
---
paper_title: Incorporating camera metadata for attended region detection and consumer photo classification

paper_content:

Photos taken by human beings significantly differ from the pictures that are taken by a surveillance camera or a vision sensor on a robot, e.g., human beings may intentionally capture photos to express his/her feeling or record a memorial scene. Such a creative photo capture process is accomplished by adjusting two factors: (1) the parameters setting of a camera; and (2) the position between the camera and the interesting objects or scenes. To enable automatic understanding and interpretation of the semantics of photos, it is very important to take all these factors into account. Unfortunately, most existing algorithms for image understanding focus on only the content of the images while completely ignoring these two important factors. In this paper, we have developed a new algorithm to calculate what the interestingness of the photographer is and what the core content of a photo is. The gained information (i.e., attended regions and attention of the photographer) is further used to support more effective photo classification and retrieval. Our experiments on 70,000+ photos taken by 200+ different models of cameras have obtained very positive results.
---
paper_title: Image Annotation Within the Context of Personal Photo Collections Using Hierarchical Event and Scene Models

paper_content:

Most image annotation systems consider a single photo at a time and label photos individually. In this work, we focus on collections of personal photos and exploit the contextual information naturally implied by the associated GPS and time metadata. First, we employ a constrained clustering method to partition a photo collection into event-based subcollections, considering that the GPS records may be partly missing (a practical issue). We then use conditional random field (CRF) models to exploit the correlation between photos based on 1) time-location constraints and 2) the relationship between collection-level annotation (i.e., events) and image-level annotation (i.e., scenes). With the introduction of such a multilevel annotation hierarchy, our system addresses the problem of annotating consumer photo collections that requires a more hierarchical description of the customers' activities than do the simpler image annotation tasks. The efficacy of the proposed system is validated by extensive evaluation using a sizable geotagged personal photo collection database, which consists of over 100 photo collections and is manually labeled for 12 events and 12 scenes to create ground truth.
---
paper_title: Consumer image retrieval by estimating relation tree from family photo collections

paper_content:

In this paper, we propose an approach to automatically estimate relationship among people in a family image collection based on results from face analyses technologies including automated face recognition and clustering, demographic assessment, and face similarity measurement, as well as contextual information such as people co-appearance, people's relative positions in photos and image timestamps. As the result, a relation tree can be estimated which provides important semantic information regarding people involved in a photo collection and has numerous applications in photo sharing and browsing, social networking, etc. The methods for deriving and integrating information from photos and the process for estimating a relation tree are described. Experimental results on two typical consumer photo collections and examples of using these results in consumer image retrieval are presented.
---
paper_title: Personal Photo Organizer Based on Automated Annotation Framework

paper_content:

Current researches toward solving personal photo management suffered two problems: (1) lacking of training data, and (2) no consolidated reference for classification. In this paper, we propose an automated annotation framework to address these problems. The framework was composed by three main components: the context information generator, the semantic concept detector, and the face recognition model. By assigning multi-labels for each photo, the framework makes the photo collection more structured and searchable. Our experimental results show that the techniques used in this framework are promising.
---
paper_title: Textual Query of Personal Photos Facilitated by Large-Scale Web Data

paper_content:

The rapid popularization of digital cameras and mobile phone cameras has led to an explosive growth of personal photo collections by consumers. In this paper, we present a real-time textual query-based personal photo retrieval system by leveraging millions of Web images and their associated rich textual descriptions (captions, categories, etc.). After a user provides a textual query (e.g., "water”), our system exploits the inverted file to automatically find the positive Web images that are related to the textual query "water” as well as the negative Web images that are irrelevant to the textual query. Based on these automatically retrieved relevant and irrelevant Web images, we employ three simple but effective classification methods, k-Nearest Neighbor (kNN), decision stumps, and linear SVM, to rank personal photos. To further improve the photo retrieval performance, we propose two relevance feedback methods via cross-domain learning, which effectively utilize both the Web images and personal images. In particular, our proposed cross-domain learning methods can learn robust classifiers with only a very limited amount of labeled personal photos from the user by leveraging the prelearned linear SVM classifiers in real time. We further propose an incremental cross-domain learning method in order to significantly accelerate the relevance feedback process on large consumer photo databases. Extensive experiments on two consumer photo data sets demonstrate the effectiveness and efficiency of our system, which is also inherently not limited by any predefined lexicon.
---
paper_title: Fast near duplicate detection for personal image collections

paper_content:

Due to the rapid growth in personal image collections, there is increasing interest on automatic detection of near duplicates. In this paper, we propose a novel fast near duplicate detection framework that takes advantages of heterogeneous features like EXIF data, global image histogram and local features. To improve the accuracy of local feature matching, we have developed a structure matching algorithm that takes into account of a local feature's neighborhood which can effectively reject mismatches. In addition, we developed a computation-sensitive cascade framework to combine stage classifiers trained on different feature spaces with different computational cost. This method can quickly accept easily identified duplicates using only cheap features without the need to extract more sophisticate but expensive ones. Compared with existing approaches, our experiments show very promising results using our new approach in terms of both efficiency and effectiveness.
---
paper_title: Using context information and local feature points in face clustering for consumer photos

paper_content:

We introduce local feature points to achieve face clustering for consumer photos. After combining eigenfaces with context information like clothes, we further investigate the usage of local feature points to match face images. The relationships between face images are constructed by feature matching and then described as a graph. Outliers in the results of preliminary clustering are detected and are re-clustered according to matching characteristics. We report complete performance comparison for different datasets and show that the proposed method has superior performance than conventional approaches.
---
paper_title: Unsupervised and fast continent classification of digital image collections using time

paper_content:

Advances in storage capacity means that digital cameras can store huge collections of digital photographs. Typically such images are given non-descriptive filenames names such as a unique identifier, often an integer. Consequently it is time-consuming and difficult to browse and retrieve images from large collections especially on small consumer electronics devices. A strategy for classifying images into geographical regions is presented which allows images to be coarsely sorted into the continent where they were taken. The strategy employs patterns in the time-stamps of images to identify events such as holiday and individual days, and to estimate the approximate longitude where the photographs were taken. Experimental evaluations demonstrate that the continent is correctly estimated for 89 % of the images in arbitrary collections and that the longitude is estimated with a mean error of 27.5 degrees. The strategy is relatively straightforward to implement, also in hardware, and computationally inexpensive.
---
paper_title: A framework for photo-quality assessment and enhancement based on visual aesthetics

paper_content:

We present an interactive application that enables users to improve the visual aesthetics of their digital photographs using spatial recomposition. Unlike earlier work that focuses either on photo quality assessment or interactive tools for photo editing, we enable the user to make informed decisions about improving the composition of a photograph and to implement them in a single framework. Specifically, the user interactively selects a foreground object and the system presents recommendations for where it can be moved in a manner that optimizes a learned aesthetic metric while obeying semantic constraints. For photographic compositions that lack a distinct foreground object, our tool provides the user with cropping or expanding recommendations that improve its aesthetic quality. We learn a support vector regression model for capturing image aesthetics from user data and seek to optimize this metric during recomposition. Rather than prescribing a fully-automated solution, we allow user-guided object segmentation and inpainting to ensure that the final photograph matches the user's criteria. Our approach achieves 86% accuracy in predicting the attractiveness of unrated images, when compared to their respective human rankings. Additionally, 73% of the images recomposited using our tool are ranked more attractive than their original counterparts by human raters.
---
paper_title: Image harmony for consumer images

paper_content:

Images elicit a variety of emotional responses related to image content, overall aesthetic appeal, or a combination of both. One aspect of aesthetic appeal is harmony: the pleasing or congruent arrangement of parts producing internal calm or tranquility. We conducted a series of experiments to identify what low level features could predict harmony in an image. Subjective judgments of image harmony were collected for images representative of typical consumer photography. Our initial results show that for simplified images (pixelated to control for emotional responses to scenes and objects) impressions of image harmony depend on statistical properties of low level local features. The feature combinations vary for different individuals but typically involve edge contrast, average lightness and range of lightness. At the same time inclusion of Gestalt principles is needed to account for the subjects' data. Additionally, global low level features related to spatial image structure may help to explain results with black and white and color images.
---
paper_title: Towards intelligent photo composition-automatic detection of unintentional dissection lines in environmental portrait photos

paper_content:

Good photos result from careful attention to various elements such as shutter speed, aperture, light exposure and composition. Over the years, there have been interests to develop digital still cameras with ''composition advising'' functions. However, not many composition-related functions have been realized in current digital cameras. This study proposed an approach for automatic detection of unintentional dissection lines which often degrade the aesthetics of an environmental portrait photo. The algorithm includes modules for face detection, ROI estimation, morphology filtering, edge detection and straight line detection. Experimental evaluations conducted to verify the performance of the algorithm show that the detection rate and false detection rate of general dissection lines are 80.87% and 33.61%, respectively.
---
paper_title: A visual attention model for adapting images on small displays

paper_content:

Image adaptation, one of the essential problems in adaptive content delivery for universal access, has been actively explored for some time. Most existing approaches have focused on generic adaptation with a view to saving file size under constraints in client environment and have hardly paid attention to user perceptions of the adapted result. Meanwhile, the major limitation on the user's delivery context is moving away from data volume (or time-to-wait) to screen size because of the galloping development of hardware technologies. In this paper, we propose a novel method for adapting images based on user attention. A generic and extensible image attention model is introduced based on three attributes (region of interest, attention value, and minimal perceptible size) associated with each attention object. A set of automatic modeling methods are presented to support this approach. A branch-and-bound algorithm is also developed to find the optimal adaptation efficiently. Experimental results demonstrate the usefulness of the proposed scheme and its potential application in the future.
---
paper_title: Unsupervised automation of photographic composition rules in digital still cameras

paper_content:

When taking pictures, professional photographers apply photographic composition rules, e.g. rule of thirds. The rule of thirds says to place the main subject's center at one of four places: at 1/3 or 2/3 of the picture width from left edge, and 1/3 or 2/3 of the picture height from the top edge. This paper develops low-complexity unsupervised methods for digital still cameras to (1) segment the main subject and (2) realize the rule-of-thirds. ::: ::: The main subject segmentation method uses the auto-focus filter, opens the shutter aperture fully, and segments the resulting image. These camera settings place the main subject in focus and blur the rest of the image by diffused light. The segmentation utilizes the difference in frequency content between the main subject and blurred background. The segmentation does not depend on prior knowledge of the indoor/outdoor setting or scene content. ::: ::: The rule-of-thirds method moves the centroid of the main subject to the closest of the four rule-of-thirds locations. We first define an objective function that measures how close the main subject placement obeys the rule-of-thirds, and then reposition the main subject in order to optimize the objective function. For multiple main subjects, the proposed algorithm could be extended to use rule-of-triangles by adding an appropriate constraint.
---
paper_title: Content-Based Photo Quality Assessment

paper_content:

Automatically assessing photo quality from the perspective of visual aesthetics is of great interest in high-level vision research and has drawn much attention in recent years. In this paper, we propose content-based photo quality assessment using regional and global features. Under this framework, subject areas, which draw the most attentions of human eyes, are first extracted. Then regional features extracted from subject areas and the background regions are combined with global features to assess the photo quality. Since professional photographers may adopt different photographic techniques and may have different aesthetic criteria in mind when taking different types of photos (e.g. landscape versus portrait), we propose to segment regions and extract visual features in different ways according to the categorization of photo content. Therefore we divide the photos into seven categories based on their content and develop a set of new subject area extraction methods and new visual features, which are specially designed for different categories. This argument is supported by extensive experimental comparisons of existing photo quality assessment approaches as well as our new regional and global features over different categories of photos. Our new features significantly outperform the state-of-the-art methods. Another contribution of this work is to construct a large and diversified benchmark database for the research of photo quality assessment. It includes 17, 613 photos with manually labeled ground truth.
---
paper_title: Classifying Images Collected on the World Wide Web

paper_content:

This work presents the classification of images collected on the World Wide Web, using a supervised classification method, called ID3 (Itemized Dichotomizer 3). The classification consists in separating the images into two semantic classes: graphics and photographs. Photographs include natural scenes, like people, faces, animals, flowers, landscapes and cities. Graphics are logos, drawings, icons, maps, and backgrounds, usually generated by computer. To validate the classifier we used the k-fold cross-validation method. In the experimental tests 95.6% of the images were correctly classified.
---
paper_title: Give and take: a study of consumer photo-sharing culture and practice

paper_content:

In this paper, we present initial findings from the study of a digital photo-sharing website: Flickr.com. In particular, we argue that Flickr.com appears to support-for some people-a different set of photography practices, socialization styles, and perspectives on privacy that are unlike those described in previous research on consumer and amateur photographers. Further, through our examination of digital photographers' photowork activities-organizing, finding, sharing and receiving-we suggest that privacy concerns and lack of integration with existing communication channels have the potential to prevent the 'Kodak Culture' from fully adopting current photo-sharing solutions.
---
paper_title: A computationally efficient approach to indoor/outdoor scene classification

paper_content:

Prior research in scene classification has shown that high-level information can be inferred from low-level image features. Classification rates of roughly 90% have been reported using low-level features to predict indoor scenes vs. outdoor scenes. However, the high classification rates are often achieved by using computationally expensive, high-dimensional feature sets, thus limiting the practical implementation of such systems. We show that a more computationally efficient approach to indoor/outdoor classification can yield classification rates comparable to the best methods reported in the literature. A low complexity, low-dimensional feature set is used in conjunction with a two-stage support vector machine classification scheme to achieve a classification rate of 90.2% on a large database of consumer photographs.
---
paper_title: Identifying personal photo digital library features

paper_content:

At present, little evidence is available about how people want to interact with their photos in a personal photo digital library. Analysis of a set of 22 user needs summaries and critiques of existing photo management systems provides insight into potentially useful features.
---
paper_title: Why did you take this photo: a study on user intentions in digital photo productions

paper_content:

Why do people take pictures? While substantial work has been done on the intentions of people producing photos in specific domains (e.g. Flickr users or mobile phone cameras) the general case - arbitrary users taking photos with arbitrary devices - has not yet been investigated in detail. In this short paper, we present an exploratory study on the intentions and goals of people taking digital photos. 40 different specific photo taking situations of 10 people where investigated. We classified the situations and reflected them into context of existing work to investigate the applicability and usefulness of classifications existing for specific domains. Our findings show that applicability of existing models for small domains is limited and they indicate the need of a generalized taxonomy. We further identify hypotheses and research questions for future work in this area.
---
paper_title: Image harmony for consumer images

paper_content:

Images elicit a variety of emotional responses related to image content, overall aesthetic appeal, or a combination of both. One aspect of aesthetic appeal is harmony: the pleasing or congruent arrangement of parts producing internal calm or tranquility. We conducted a series of experiments to identify what low level features could predict harmony in an image. Subjective judgments of image harmony were collected for images representative of typical consumer photography. Our initial results show that for simplified images (pixelated to control for emotional responses to scenes and objects) impressions of image harmony depend on statistical properties of low level local features. The feature combinations vary for different individuals but typically involve edge contrast, average lightness and range of lightness. At the same time inclusion of Gestalt principles is needed to account for the subjects' data. Additionally, global low level features related to spatial image structure may help to explain results with black and white and color images.
---
paper_title: The Design of High-Level Features for Photo Quality Assessment

paper_content:

We propose a principled method for designing high level features forphoto quality assessment. Our resulting system can classify between high quality professional photos and low quality snapshots. Instead of using the bag of low-level features approach, we first determine the perceptual factors that distinguish between professional photos and snapshots. Then, we design high level semantic features to measure the perceptual differences. We test our features on a large and diverse dataset and our system is able to achieve a classification rate of 72% on this difficult task. Since our system is able to achieve a precision of over 90% in low recall scenarios, we show excellent results in a web image search application.
---
paper_title: Region based image appeal metric for consumer photos

paper_content:

Image appeal may be defined as the interest that a photograph generates when viewed by human observers, incorporating subjective factors on top of the traditional objective quality measures. User studies were conducted in order to identify the right features to use in an image appeal measure; these studies also revealed that a photograph may be appealing even if only a region/area of the photograph is actually appealing. Extensive experimentation helped identify a good set of low level features, which were combined into a set of image appeal metrics. The appealing region detection and the image appeal metrics were validated with experiments conducted on 2000 ground truth images, each graded by three human observers. Two main image appeal metrics are presented: one that ranks the images based on image appeal with a high correlation with the human observations; and a second one which performs better at retrieving highly appealing images from an image collection.
---

Title: A Survey on Automatic Techniques for Enhancement and Analysis of Digital Photography

Section 1: Introduction
Description 1: Introduce the background, motivation, and the scope of the survey in digital photography enhancement and analysis.

Section 2: Methodology of the Research
Description 2: Present the methodology adopted for searching related work, detailing search engines, digital libraries, and keywords used in the process.

Section 3: Breadth-First Search
Description 3: Explain the breadth-first search strategy used to find related papers within recent technical publications.

Section 4: Depth-First Search
Description 4: Detail the depth-first search methodology, including the use of keywords and referenced papers.

Section 5: Search Results
Description 5: Present the results from the search strategies and discuss relevant outcomes, including conferences and journals identified.

Section 6: Considerations on the Methodology
Description 6: Discuss the considerations and limitations of the methodology adopted for the survey.

Section 7: Enhancement
Description 7: Review research on enhancement techniques applied to digital photography, separating on the fly and off-line approaches.

Section 8: On the Fly Enhancement
Description 8: Discuss specific on the fly enhancement techniques, including mechanisms embedded in modern digital cameras.

Section 9: Off-Line Enhancement
Description 9: Examine methods for the enhancement of photos that have already been acquired, discussing techniques such as image inpainting, cropping, and recomposition.

Section 10: Analysis
Description 10: Present relevant work on photography analysis, organizing methods according to goals like assessment and information extraction.

Section 11: Assessment
Description 11: Detail assessment algorithms that score photos based on various metrics related to image quality and aesthetics.

Section 12: Information Extraction
Description 12: Discuss approaches for extracting elements of interest important to photographic analysis systems, such as face detection and landscape analysis.

Section 13: Grouping
Description 13: Describe methods for photo grouping, including classification, clustering, summarization, and image retrieval.

Section 14: Discussion
Description 14: Summarize and organize the algorithms for photo analysis into three categories: assessment, information extraction, and grouping.

Section 15: Critical Analysis
Description 15: Provide a critical analysis covering the main issues found in the reviewed approaches, including image sets and validation methods.

Section 16: Image Sets
Description 16: Review the sources and sizes of image sets used in the surveyed studies and discuss related challenges and conclusions.

Section 17: Validation
Description 17: Discuss the methods used to validate photo enhancement and analysis approaches, comparing objective and subjective validation methods.

Section 18: Conclusions
Description 18: Summarize the main findings of the survey, discuss the current state-of-the-art, and highlight areas with potential for future research improvements.