A Survey on Video Inpainting
---
paper_title: Video Inpainting Under Constrained Camera Motion

paper_content:

A framework for inpainting missing parts of a video sequence recorded with a moving or stationary camera is presented in this work. The region to be inpainted is general: it may be still or moving, in the background or in the foreground, it may occlude one object and be occluded by some other object. The algorithm consists of a simple preprocessing stage and two steps of video inpainting. In the preprocessing stage, we roughly segment each frame into foreground and background. We use this segmentation to build three image mosaics that help to produce time consistent results and also improve the performance of the algorithm by reducing the search space. In the first video inpainting step, we reconstruct moving objects in the foreground that are "occluded" by the region to be inpainted. To this end, we fill the gap as much as possible by copying information from the moving foreground in other frames, using a priority-based scheme. In the second step, we inpaint the remaining hole with the background. To accomplish this, we first align the frames and directly copy when possible. The remaining pixels are filled in by extending spatial texture synthesis techniques to the spatiotemporal domain. The proposed framework has several advantages over state-of-the-art algorithms that deal with similar types of data and constraints. It permits some camera motion, is simple to implement, fast, does not require statistical models of background nor foreground, works well in the presence of rich and cluttered backgrounds, and the results show that there is no visible blurring or motion artifacts. A number of real examples taken with a consumer hand-held camera are shown supporting these findings.
---
paper_title: Video completion using tracking and fragment merging

paper_content:

Video completion is the problem of automatically filling space–time holes in video sequences left by the removal of unwanted objects in a scene. We solve it using texture synthesis, filling a hole inwards using three steps iteratively: we select the most promising target pixel at the edge of the hole, we find the source fragment most similar to the known part of the target’s neighborhood, and we merge source and target fragments to complete the target neighborhood, reducing the size of the hole.
---
paper_title: Video epitomes

paper_content:

Recently, "epitomes" were introduced as patch-based probability models that are learned by compiling together a large number of examples of patches from input images. In this paper, we describe how epitomes can be used to model video data and we describe significant computational speedups that can be incorporated into the epitome inference and learning algorithm. In the case of videos, epitomes are estimated so as to model most of the small space-time cubes from the input data. Then, the epitome can be used for various modeling and reconstruction tasks, of which we show results for video super-resolution, video interpolation, and object removal. Besides computational efficiency, an interesting advantage of the epitome as a representation is that it can be reliably estimated even from videos with large amounts of missing data. We illustrate this ability on the task of reconstructing the dropped frames in video broadcast using only the degraded video.
---
paper_title: Contour-Based Video Inpainting

paper_content:

NA
---
paper_title: Video Completion for Perspective Camera Under Constrained Motion

paper_content:

This paper presents a novel technique to fill in missing background and moving foreground of a video captured by a static or moving camera. Different from previous efforts which are typically based on processing in the 3D data volume, we slice the volume along the motion manifold of the moving object, and therefore reduce the search space from 3D to 2D, while still preserve the spatial and temporal coherence. In addition to the computational efficiency, based on geometric video analysis, the proposed approach is also able to handle real videos under perspective distortion, as well as common camera motions, such as panning, tilting, and zooming. The experimental results demonstrate that our algorithm performs comparably to 3D search based methods, and however extends the current state-of-the-art repairing techniques to videos with projective effects, as well as illumination changes.
---
paper_title: Space-Time Completion of Video

paper_content:

This paper presents a new framework for the completion of missing information based on local structures. It poses the task of completion as a global optimization problem with a well-defined objective function and derives a new algorithm to optimize it. Missing values are constrained to form coherent structures with respect to reference examples. We apply this method to space-time completion of large space-time "holes" in video sequences of complex dynamic scenes. The missing portions are filled in by sampling spatio-temporal patches from the available parts of the video, while enforcing global spatio-temporal consistency between all patches in and around the hole. The consistent completion of static scene parts simultaneously with dynamic behaviors leads to realistic looking video sequences and images. Space-time video completion is useful for a variety of tasks, including, but not limited to: 1) sophisticated video removal (of undesired static or dynamic objects) by completing the appropriate static or dynamic background information. 2) Correction of missing/corrupted video frames in old movies. 3) Modifying a visual story by replacing unwanted elements. 4) Creation of video textures by extending smaller ones. 5) Creation of complete field-of-view stabilized video. 6) As images are one-frame videos, we apply the method to this special case as well
---
paper_title: A Rank Minimization Approach to Video Inpainting

paper_content:

This paper addresses the problem of video inpainting, that is seamlessly reconstructing missing portions in a set of video frames. We propose to solve this problem proceeding as follows: (i) finding a set of descriptors that encapsulate the information necessary to reconstruct a frame, (ii) finding an optimal estimate of the value of these descriptors for the missing/corrupted frames, and (iii) using the estimated values to reconstruct the frames. The main result of the paper shows that the optimal descriptor estimates can be efficiently obtained by minimizing the rank of a matrix directly constructed from the available data, leading to a simple, computationally attractive, dynamic inpainting algorithm that optimizes the use of spatio/temporal information. Moreover, contrary to most currently available techniques, the method can handle non-periodic target motions, non-stationary backgrounds and moving cameras. These results are illustrated with several examples, including reconstructing dynamic textures and object disocclusion in cases involving both moving targets and camera.
---
paper_title: Video repairing: inference of foreground and background under severe occlusion

paper_content:

We propose a new method, video repairing, to robustly infer missing static background and moving foreground due to severe damage or occlusion from a video. To recover background pixels, we extend the image repairing method, where layer segmentation and homography blending are used to preserve temporal coherence and avoid flickering. By exploiting the constraint imposed by periodic motion and a subclass of camera and object motions, we adopt a two-phase approach to repair moving foreground pixels: In the sampling phase, motion data are sampled and regularized by 3D tensor voting to maintain temporal coherence and motion periodicity. In the alignment phase, missing moving foreground pixels are inferred by spatial and temporal alignment of the sampled motion data at multiple scales. We experimented our system with some difficult examples, where the camera can be stationary or in motion.
---
paper_title: Human Object Inpainting Using Manifold Learning-Based Posture Sequence Estimation

paper_content:

We propose a human object inpainting scheme that divides the process into three steps: 1) human posture synthesis; 2) graphical model construction; and 3) posture sequence estimation. Human posture synthesis is used to enrich the number of postures in the database, after which all the postures are used to build a graphical model that can estimate the motion tendency of an object. We also introduce two constraints to confine the motion continuity property. The first constraint limits the maximum search distance if a trajectory in the graphical model is discontinuous, and the second confines the search direction in order to maintain the tendency of an object's motion. We perform both forward and backward predictions to derive local optimal solutions. Then, to compute an overall best solution, we apply the Markov random field model and take the potential trajectory with the maximum total probability as the final result. The proposed posture sequence estimation model can help identify a set of suitable postures from the posture database to restore damaged/missing postures. It can also make a reconstructed motion sequence look continuous.
---
paper_title: Efficient Object-Based Video Inpainting

paper_content:

Video inpainting describes the process of removing a portion of a video and filling in the missing part (hole) in a visually consistent manner. Most existing video inpainting techniques are computationally intensive and cannot handle large holes. In this paper, we propose a complete and efficient video in-painting system. Our system applies different strategies to handle static and dynamic portions of the hole. To inpaint the static portion, our system uses background replacement and image inpainting techniques. To inpaint moving objects in the hole, we utilizes background subtraction and object segmentation to extract a set of object templates and perform optimal object interpolation using dynamic programming. We evaluate the performance of our system based on a set of indoor surveillance sequences with different types of occlusions.
---
paper_title: Video repairing under variable illumination using cyclic motions

paper_content:

This paper presents a complete system capable of synthesizing a large number of pixels that are missing due to occlusion or damage in an uncalibrated input video. These missing pixels may correspond to the static background or cyclic motions of the captured scene. Our system employs user-assisted video layer segmentation, while the main processing in video repair is fully automatic. The input video is first decomposed into the color and illumination videos. The necessary temporal consistency is maintained by tensor voting in the spatio-temporal domain. Missing colors and illumination of the background are synthesized by applying image repairing. Finally, the occluded motions are inferred by spatio-temporal alignment of collected samples at multiple scales. We experimented on our system with some difficult examples with variable illumination, where the capturing camera can be stationary or in motion.
---

Title: A Survey on Video Inpainting

Section 1: INTRODUCTION
Description 1: This section introduces the concept of inpainting and its applications, specifically focusing on the differences and challenges between image and video inpainting.

Section 2: PATCH BASED TECHNIQUES
Description 2: This section describes various patch-based video inpainting methods, their evolution from image inpainting techniques, and their advantages and limitations.

Section 3: OBJECT BASED TECHNIQUES
Description 3: This section discusses object-based approaches to video inpainting, which consider both spatial and temporal consistencies, and highlights different algorithms, their benefits, and drawbacks.

Section 4: COMPARISON OF VARIOUS ALGORITHMS
Description 4: This section compares the merits and demerits of different video inpainting algorithms covered in the previous sections, focusing on aspects like computational complexity, artifact generation, and suitability for different types of motions and scenes.

Section 5: CONCLUSION
Description 5: This section summarizes the survey, emphasizing the differences between patch-based and object-based algorithms, and discusses potential guidelines for choosing an appropriate algorithm based on specific requirements.