Image and Video Compression with Neural Networks: A Review
---
paper_title: Motion Vector Coding in the HEVC Standard

paper_content:

High Efficiency Video Coding (HEVC) is an emerging international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). Compared to H.264/AVC, HEVC has achieved substantial compression performance improvement. During the HEVC standardization, we proposed several motion vector coding techniques, which were crosschecked by other experts and then adopted into the standard. In this paper, an overview of the motion vector coding techniques in HEVC is firstly provided. Next, the proposed motion vector coding techniques including a priority-based derivation algorithm for spatial motion candidates, a priority-based derivation algorithm for temporal motion candidates, a surrounding-based candidate list, and a parallel derivation of the candidate list, are also presented. Based on HEVC test model 9 (HM9), experimental results show that the combination of the proposed techniques achieves on average 3.1% bit-rate saving under the common test conditions used for HEVC development.
---
paper_title: Nonlocal In-Loop Filter: The Way Toward Next-Generation Video Coding?

paper_content:

In-loop filtering has emerged as an essential coding tool since H.264/AVC, due to its delicate design, which reduces different kinds of compression artifacts. However, existing in-loop filters rely only on local image correlations, largely ignoring nonlocal similarities. In this article, the authors explore the design philosophy of in-loop filters and discuss their vision for the future of in-loop filter research by examining the potential of nonlocal similarities. Specifically, the group-based sparse representation, which jointly exploits an image's local and nonlocal self-similarities, lays a novel and meaningful groundwork for in-loop filter design. Hard- and soft-thresholding filtering operations are applied to derive the sparse parameters that are appropriate for compression artifact reduction. Experimental results show that this in-loop filter design can significantly improve the compression performance of the High Efficiency Video Coding (HEVC) standard, leading us in a new direction for improving compression efficiency.
---
paper_title: Arithmetic coding for data compression

paper_content:

The state of the art in data compression is arithmetic coding, not the better-known Huffman method. Arithmetic coding gives greater compression, is faster for adaptive models, and clearly separates the model from the channel encoding.
---
paper_title: Low-Rank-Based Nonlocal Adaptive Loop Filter for High-Efficiency Video Compression

paper_content:

In video coding, the in-loop filtering has emerged as a key module due to its significant improvement on compression performance since H.264/Advanced Video Coding. Existing incorporated in-loop filters in video coding standards mainly take advantage of the local smoothness prior model used for images. In this paper, we propose a novel adaptive loop filter utilizing image nonlocal prior knowledge by imposing the low-rank constraint on similar image patches for compression noise reduction. In the filtering process, the reconstructed frame is first divided into image patch groups according to image patch similarity. The proposed in-loop filtering is formulated as an optimization problem with low-rank constraint for every group of image patches independently. It can be efficiently solved by soft-thresholding singular values of the matrix composed of image patches in the same group. To adapt the properties of the input sequences and bit budget, an adaptive threshold derivation model is established for every group of image patches according to the characteristics of compressed image patches, quantization parameters, and coding modes. Moreover, frame-level and largest coding unit-level control flags are signaled to further improve the adaptability from the sense of rate-distortion optimization. The performance of the proposed in-loop filter is analyzed when it collaborates with the existing in-loop filters in High Efficiency Video Coding. Extensive experimental results show that our proposed in-loop filter can further improve the performance of state-of-the-art video coding standard significantly, with up to 16% bit-rate savings.
---
paper_title: Hadamard transform image coding

paper_content:

The introduction of the fast Fourier transform algorithm has led to the development of the Fourier transform image coding technique whereby the two-dimensional Fourier transform of an image is transmitted over a channel rather than the image itself. This devlopement has further led to a related image coding technique in which an image is transformed by a Hadamard matrix operator. The Hadamard matrix is a square array of plus and minus ones whose rows and columns are orthogonal to one another. A high-speed computational algorithm, similar to the fast Fourier transform algorithm, which performs the Hadamard transformation has been developed. Since only real number additions and subtractions are required with the Hadamard transform, an order of magnitude speed advantage is possible compared to the complex number Fourier transform. Transmitting the Hadamard transform of an image rather than the spatial representation of the image provides a potential toleration to channel errors and the possibility of reduced bandwidth transmission.
---
paper_title: Discrete Cosine Transform

paper_content:

A discrete cosine transform (DCT) is defined and an algorithm to compute it using the fast Fourier transform is developed. It is shown that the discrete cosine transform can be used in the area of digital processing for the purposes of pattern recognition and Wiener filtering. Its performance is compared with that of a class of orthogonal transforms and is found to compare closely to that of the Karhunen-Loeve transform, which is known to be optimal. The performances of the Karhunen-Loeve and discrete cosine transforms are also found to compare closely with respect to the rate-distortion criterion.
---
paper_title: Motion-compensated transform coding

paper_content:

Interframe hybrid transform/dpcm coders encode television signals by taking a spatial transform of a block of picture elements in a frame and predictively coding the resulting coefficients using the corresponding coefficients of the spatial block at the same location in the previous frame. These coders can be made more efficient for scenes containing objects in translational motion by first estimating the translational displacement of objects and then using coefficients of a spatially displaced block in the previous frame for prediction. This paper presents simulation results for such motion-compensated transform coders using two algorithms for estimating displacements. The first algorithm, which is developed in a companion paper, recursively estimates the displacements from the previously transmitted transform coefficients, thereby eliminating the need to transmit the displacement estimates. The second algorithm, due to Limb and Murphy, estimates displacements by taking ratios of accummulated frame difference and spatial difference signals in a block. In this scheme, the displacement estimates are transmitted to the receiver. Computer simulations on two typical real-life sequences of frames show that motion-compensated coefficient prediction results in coder bit rates that are 20 to 40 percent tower than conventional interframe transform coders using “frame difference of coefficients.” Comparisons of bit rates for approximately the same picture quality show that the two methods of displacement estimation are quite similar in performance with a slight preference for the scheme with recursive displacement estimation.
---
paper_title: Adaptive loop filter with temporal prediction

paper_content:

In this paper, we propose a method to improve adaptive loop filter (ALF) efficiency with temporal prediction. For one frame, two sets of adaptive loop filter parameters are adaptively selected by rate distortion optimization. The first set of ALF parameters is estimated by minimizing the mean square error between the original frame and the current reconstructed frame. The second set of filter parameters is the one that is used in the latest prior frame. The proposed algorithm is implemented in HM3.0 software. Compared with the HM3.0 anchor, the proposed method achieves 0.4%, 0.3% and 0.3% BD bitrate reduction in average for high efficiency low delay B, high efficiency low delay P and high efficiency random access configuration, respectively. The encoding and decoding time increase by 1% and 2% on average, respectively.
---
paper_title: Adaptive bilateral filter for improved in-loop filtering in the emerging high efficiency video coding standard

paper_content:

To face the still growing video compression needs, ITU and MPEG have jointly started a standardization project called High Efficiency Video Coding (HEVC) which aims to improve the compression efficiency of the state-of-the-art H.264/AVC standard for high and ultra high definition video. The HEVC codec still relies on the usual motion compensated, predictive, block based transform coding architecture with block sizes higher than 8×8 being now considered. Furthermore, the codec is also equipped with a Wiener in-loop filter which, together with the H.264/AVC deblocking filter, further reduces the distortion between the original and decoded frames introduced by lossy coding. While this filter allows indeed reducing the sum of square differences between the original and reconstructed frames, it is not specifically designed to reduce the ringing artifacts resulting from the use of large transform block sizes. In this context, this paper proposes to combine an adaptive bilateral filter together with the HEVC Wiener filter. The proposed combined filter allows an average bitrate reduction of about 7% regarding the HEVC codec without the Wiener filter and a 1.5% reduction regarding the HEVC codec with only the Wiener filter, always for the same quality. Moreover, the combined filter also reduces the ringing artifacts according to an objective metric specifically designed to quantify this type of coding artifact.
---
paper_title: HEVC Deblocking Filter

paper_content:

This paper describes the in-loop deblocking filter used in the upcoming High Efficiency Video Coding (HEVC) standard to reduce visible artifacts at block boundaries. The deblocking filter performs detection of the artifacts at the coded block boundaries and attenuates them by applying a selected filter. Compared to the H.264/AVC deblocking filter, the HEVC deblocking filter has lower computational complexity and better parallel processing capabilities while still achieving significant reduction of the visual artifacts.
---
paper_title: A comparison of fractional-pel interpolation filters in HEVC and H.264/AVC

paper_content:

The fractional-pel interpolation filter adopted in H.264/AVC improves motion compensation greatly. Recently, a new DCT-based fractional-pel interpolation filter is adopted in the oncoming standard HEVC. We are interested in the differences between these two types of fractional-pel interpolation filters. In this paper we describe the derivations of fractional-pel interpolation filters in HEVC and H.264/AVC in detail, and compare them on properties of frequency responses. We find that the half-pel interpolation filters in HEVC and H.264/AVC are very similar, but the low-pass properties of quarter-pel interpolation filters in HEVC are much better than those in H.264/AVC. Experimental results validate this phenomenon, the fractional-pel interpolation in H.264/AVC tends to increase BD-rates by more than 10% compared with that in HEVC, and this performance loss mainly comes from quarter-pel interpolation filters. On the other hand, the complexity of fractional-pel interpolation filtering in HEVC is greatly increased than that in H.264/AVC.
---
paper_title: Experiments with linear prediction in television

paper_content:

The correlation present in a signal makes possible the prediction of the future of the signal in terms of the past and present. If the method used for prediction makes full use of the entire pertinent past, then the error signal — the difference between the actual and the predicted signal — will be a completely random wave of lower power than the original signal but containing all the information of the original. One method of prediction, which docs not make full use of the past, but which is nevertheless remarkably effective with certain signals and also appealing because of its relative simplicity, is linear prediction. Here the prediction for the next signal sample is simply the sum of previous signal samples each multiplied by an appropriate weighting factor. The best values for the weighting coefficients depend upon the statistics of the signal, but once they have been determined the prediction may be done with relatively simple apparatus. This paper describes the apparatus used for some experiments on linear prediction of television signals, and describes the results obtained to date.
---
paper_title: Sample Adaptive Offset in the HEVC Standard

paper_content:

This paper provides a technical overview of a newly added in-loop filtering technique, sample adaptive offset (SAO), in High Efficiency Video Coding (HEVC). The key idea of SAO is to reduce sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category. The offset of each category is properly calculated at the encoder and explicitly signaled to the decoder for reducing sample distortion effectively, while the classification of each sample is performed at both the encoder and the decoder for saving side information significantly. To achieve low latency of only one coding tree unit (CTU), a CTU-based syntax design is specified to adapt SAO parameters for each CTU. A CTU-based optimization algorithm can be used to derive SAO parameters of each CTU, and the SAO parameters of the CTU are inter leaved into the slice data. It is reported that SAO achieves on average 3.5% BD-rate reduction and up to 23.5% BD-rate reduction with less than 1% encoding time increase and about 2.5% decoding time increase under common test conditions of HEVC reference software version 8.0.
---
paper_title: Adaptive deblocking filter

paper_content:

This paper describes the adaptive deblocking filter used in the H.264/MPEG-4 AVC video coding standard. The filter performs simple operations to detect and analyze artifacts on coded block boundaries and attenuates those by applying a selected filter.
---
paper_title: Adaptive Loop Filtering for Video Coding

paper_content:

Adaptive loop filtering for video coding is to minimize the mean square error between original samples and decoded samples by using Wiener-based adaptive filter. The proposed ALF is located at the last processing stage for each picture and can be regarded as a tool to catch and fix artifacts from previous stages. The suitable filter coefficients are determined by the encoder and explicitly signaled to the decoder. In order to achieve better coding efficiency, especially for high resolution videos, local adaptation is used for luma signals by applying different filters to different regions or blocks in a picture. In addition to filter adaptation, filter on/off control at coding tree unit (CTU) level is also helpful for improving coding efficiency. Syntax-wise, filter coefficients are sent in a picture level header called adaptation parameter set, and filter on/off flags of CTUs are interleaved at CTU level in the slice data. This syntax design not only supports picture level optimization but also achieves a low encoding latency. Simulation results show that the ALF can achieve on average 7% bit rate reduction for 25 HD sequences. The run time increases are 1% and 10% for encoders and decoders, respectively, without special attention to optimization in C++ code.
---
paper_title: High performance scalable image compression with EBCOT

paper_content:

A new image compression algorithm is proposed, based on independent embedded block coding with optimized truncation of the embedded bit-streams (EBCOT). The algorithm exhibits state-of-the-art compression performance while producing a bit-stream with a rich feature set, including resolution and SNR scalability together with a random access property. The algorithm has modest complexity and is extremely well suited to applications involving remote browsing of large compressed images. The algorithm lends itself to explicit optimization with respect to MSE as well as more realistic psychovisual metrics, capable of modeling the spatially varying visual masking phenomenon.
---
paper_title: Overview of the High Efficiency Video Coding (HEVC) Standard

paper_content:

High Efficiency Video Coding (HEVC) is currently being prepared as the newest video coding standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group. The main goal of the HEVC standardization effort is to enable significantly improved compression performance relative to existing standards-in the range of 50% bit-rate reduction for equal perceptual video quality. This paper provides an overview of the technical features and characteristics of the HEVC standard.
---
paper_title: Intra Coding of the HEVC Standard

paper_content:

This paper provides an overview of the intra coding techniques in the High Efficiency Video Coding (HEVC) standard being developed by the Joint Collaborative Team on Video Coding (JCT-VC). The intra coding framework of HEVC follows that of traditional hybrid codecs and is built on spatial sample prediction followed by transform coding and postprocessing steps. Novel features contributing to the increased compression efficiency include a quadtree-based variable block size coding structure, block-size agnostic angular and planar prediction, adaptive pre- and postfiltering, and prediction direction-based transform coefficient scanning. This paper discusses the design principles applied during the development of the new intra coding methods and analyzes the compression performance of the individual tools. Computational complexity of the introduced intra prediction algorithms is analyzed both by deriving operational cycle counts and benchmarking an optimized implementation. Using objective metrics, the bitrate reduction provided by the HEVC intra coding over the H.264/advanced video coding reference is reported to be 22% on average and up to 36%. Significant subjective picture quality improvements are also reported when comparing the resulting pictures at fixed bitrate.
---
paper_title: Learning translation invariant recognition in a massively parallel networks

paper_content:

One major goal of research on massively parallel networks of neuron-like processing elements is to discover efficient methods for recognizing patterns. Another goal is to discover general learning procedures that allow networks to construct the internal representations that are required for complex tasks. This paper describes a recently developed procedure that can learn to perform a recognition task. The network is trained on examples in which the input vector represents an instance of a pattern in a particular position and the required output vector represents its name. After prolonged training, the network develops canonical internal representations of the patterns and it uses these canonical representations to identify familiar patterns in novel positions.
---
paper_title: Handwritten zip code recognition with multilayer networks

paper_content:

An application of back-propagation networks to handwritten zip code recognition is presented. Minimal preprocessing of the data is required, but the architecture of the network is highly constrained and specifically designed for the task. The input of the network consists of size-normalized images of isolated digits. The performance on zip code digits provided by the US Postal Service is 92% recognition, 1% substitution, and 7% rejects. Structured neural networks can be viewed as statistical methods with structure which bridge the gap between purely statistical and purely structural methods. >
---
paper_title: The JPEG still picture compression standard

paper_content:

For the past few years, a joint ISO/CCITT committee known as JPEG (Joint Photographic Experts Group) has been working to establish the first international compression standard for continuous-tone still images, both grayscale and color. JPEG’s proposed standard aims to be generic, to support a wide variety of applications for continuous-tone images. To meet the differing needs of many applications, the JPEG standard includes two basic compression methods, each with various modes of operation. A DCT-based method is specified for “lossy’’ compression, and a predictive method for “lossless’’ compression. JPEG features a simple lossy technique known as the Baseline method, a subset of the other DCT-based modes of operation. The Baseline method has been by far the most widely implemented JPEG method to date, and is sufficient in its own right for a large number of applications. This article provides an overview of the JPEG standard, and focuses in detail on the Baseline method.
---
paper_title: A neural network approach to transform image coding

paper_content:

A neural network approach is presented for transform image coding. It is shown that the three steps in the conventional transform image coding, i.e. the unitary transform of spatial domain image data, the quantization of the transform domain data and the binary coding of the quantized data, can be unified into a one-step optimization problem. Then, the optimization problem is solved by an appropriately constructed Hopfield neural network whose input is the spatial domain image data and whose output is binary codes. A practical circuit implementation is given to perform the transform image coding. the circuit has rM2 neurons, where r is the bit-rate, in bit/pixel, of the coding and M2 is the size of the images. Each neuron consists of only a non-linear voltage amplifier, a linear voltage-controlled current source, a d.c. current source, a linear passive resistor, a linear passive capacitor, and a weighted voltage summer which can be made of a single op amp with some linear passive resistors. Moreover, each neuron is locally connected with no more than b - 1 other neurons by wires, where b is the maximum bit allocated to a transform domain coefficient. Therefore, our proposed approach is particularly suitable for low-bit-rate image coding and VLSI implementation. Furthermore, the analogue and parallel nature of our approach matches perfectly the high-speed requirement of real-time image coding.
---
paper_title: Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression

paper_content:

A three-layered neural network is described for transforming two-dimensional discrete signals into generalized nonorthogonal 2-D Gabor representations for image analysis, segmentation, and compression. These transforms are conjoint spatial/spectral representations, which provide a complete image description in terms of locally windowed 2-D spectral coordinates embedded within global 2-D spatial coordinates. In the present neural network approach, based on interlaminar interactions involving two layers with fixed weights and one layer with adjustable weights, the network finds coefficients for complete conjoint 2-D Gabor transforms without restrictive conditions. In wavelet expansions based on a biologically inspired log-polar ensemble of dilations, rotations, and translations of a single underlying 2-D Gabor wavelet template, image compression is illustrated with ratios up to 20:1. Also demonstrated is image segmentation based on the clustering of coefficients in the complete 2-D Gabor transform. >
---
paper_title: Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences

paper_content:

Artificial neural networks are appearing as useful alternatives to traditional statistical modelling techniques in many scientific disciplines. This paper presents a general introduction and discussion of recent applications of the multilayer perceptron, one type of artificial neural network, in the atmospheric sciences.
---
paper_title: Neural Network Approaches to Image Compression

paper_content:

This paper presents a tutor a overview of neural networks as signal processing tools for image compression. They are well suited to the problem of image compression due to their massively parallel and distributed architecture. Their characteristics are analogous to some of the features of our own visual system, which allow us to process visual information with much ease. For example, multilayer perceptions can be used as nonlinear predictors in differential pulse-code modulation (DPCM). Such predictors have been shown to increase the predictive gain relative to a linear predictor. Another active area of research is in the application of Hebbian learning to the extraction of principal components, which are the basis vectors for the optimal linear Karhunen-Loeve transform (KLT). These learning algorithms are iterative, have some computational advantages over standard eigendecomposition techniques, and can be made to adapt to changes in the input signal. Yet another model, the self-organizing feature map (SOFM), has been used with a great deal of success in the design of codebooks for vector quantization (VQ). The resulting codebooks are less sensitive to initial conditions than the standard LBG algorithm, and the topological ordering of the entries can be exploited to further increasing the coding efficiency and reducing the computational complexity. >
---
paper_title: A non-linear predictor for differential pulse-code encoder (DPCM) using artificial neural networks

paper_content:

A nonlinear predictor is designed for a DPCM encoder using artificial neural networks (ANN). The predictor is based on a multilayer perceptron with three input nodes, 30 hidden nodes and one output node. The back-propagation learning algorithm is used for the training of the network. Simulation results are presented to evaluate and compare the performance of the neural net based predictor (nonlinear) with that of an optimized linear predictor. Success in the use of the nonlinear predictor is demonstrated through the reduction in the entropy of the differential error signal as compared to that of a linear predictor. Also it is shown that the ANN predictor is much more robust for encoding noisy images compared to that of a linear predictor. >
---
paper_title: Neural network approach to DPCM system design for image coding

paper_content:

This paper presents a neural network approach to differential pulse code modulation (DPCM) design for the encoding of images. Instead of traditional algorithms for the computation of the relevant coefficients, such as the autocovariance and autocorrelation methods, the predictor is designed by supervised training of a neural network on examples, i.e. on a typical sequence of pixel values. This allows the use of nonlinear as well as linear correlations. Efficient and fast neural net architectures, for nonlinear one-dimensional DPCM (NNDPCM) as well as two-dimensional adaptive DPCM (NNADPCM), have been designed and applied to still image coding. Computer simulation experiments have shown that the resulting encoders work very well. At a transmission rate of 1 bitpixel, the 1-D NNDPCM offers an advantage of about 4 dB in peak signal-to-noise ratio over the standard linear DPCM system. At a bit rate of 0.525 bit/pixel, the 2-D NNADPCM achieves 29.5 dB for the 512 × 512 Lena image, while there is little visible distortion in the reconstructed image. This performance is comparable to that of the best schemes known to date, whether DPCM based or not, while maintaining a lower encoding complexity. Furthermore, this establishes that there is substantial amount of nonlinear content available for 1-D and 2-D prediction in DPCM image coding.
---
paper_title: Neural model for Karhunen-Loeve transform with application to adaptive image compression

paper_content:

A neural model approach to perform adaptive calculation of the principal components (eigenvectors) of the covariance matrix of an input sequence is proposed. The algorithm is based on the successive application of the modified Hebbian learning rule proposed by Oja on every new covariance matrix that results after calculating the previous eigenvectors. The approach is shown to converge to the next dominant component that is linearly independent of all previously determined eigenvectors. The optimal learning rate is calculated by minimising an error function of the learning rate along the gradient descent direction. The approach is applied to encode grey-level images adaptively, by calculating a limited number of the KLT coefficients that meet a specified performance criterion. The effect of changing the size of the input sequence (number of image subimages), the maximum number of coding coefficients on the bit-rate values, the compression ratio, the signal-to-noise ratio, and the generalisation capability of the model to encode new images are investigated.
---
paper_title: Image compression with a hierarchical neural network

paper_content:

A neural network data compression method is presented. This network accepts a large amount of image or text data, compresses it for storage or transmission, and subsequently restores it when desired. A new training method, referred to as the Nested Training Algorithm (NTA), that reduces the training time considerably is presented. Analytical results are provided for the specification of the optimal learning rates and the size of the training data for a given image of specified dimensions. Performance of the network has been evaluated using both synthetic and real-world data. It is shown that the developed architecture and training algorithm provide high compression ratio and low distortion while maintaining the ability to generalize, and is very robust as well.
---
paper_title: A neural network approach to transform image coding

paper_content:

A neural network approach is presented for transform image coding. It is shown that the three steps in the conventional transform image coding, i.e. the unitary transform of spatial domain image data, the quantization of the transform domain data and the binary coding of the quantized data, can be unified into a one-step optimization problem. Then, the optimization problem is solved by an appropriately constructed Hopfield neural network whose input is the spatial domain image data and whose output is binary codes. A practical circuit implementation is given to perform the transform image coding. the circuit has rM2 neurons, where r is the bit-rate, in bit/pixel, of the coding and M2 is the size of the images. Each neuron consists of only a non-linear voltage amplifier, a linear voltage-controlled current source, a d.c. current source, a linear passive resistor, a linear passive capacitor, and a weighted voltage summer which can be made of a single op amp with some linear passive resistors. Moreover, each neuron is locally connected with no more than b - 1 other neurons by wires, where b is the maximum bit allocated to a transform domain coefficient. Therefore, our proposed approach is particularly suitable for low-bit-rate image coding and VLSI implementation. Furthermore, the analogue and parallel nature of our approach matches perfectly the high-speed requirement of real-time image coding.
---
paper_title: Artificial Neural Networks

paper_content:

From the Publisher: ::: This new text has been designed to present the concepts of artificial neural networks in a concise and logical manner for your computer engineering students.
---
paper_title: Artificial neural network for image compression

paper_content:

An artificial neural network is proposed which is able to compress an image by computing a nonlinear nonorthogonal transform and its inverse. The network is trained with small blocks extracted from an image; after the learning phase, it proves to give satisfactory performances both for the learned and for different unlearned images.
---
paper_title: Image data compression using a neural network model

paper_content:

Data compression and generalization capabilities are important for neural network models as learning machines. From this point of view, the image data compression characteristics of a neural network model are examined. The applied network model is a feedforward-type, three-layered network with the backpropagation learning algorithm. The implementation of this model on a hypercube parallel computer and its computation performance are described. Image data compression, generalization, and quantization characteristics are examined experimentally. Effects of learning using the discrete cosine transformation coefficients as initial connection weights are shown experimentally. >
---
paper_title: Random network learning and image compression

paper_content:

Digital image compression serves a wide range of applications. Encoding an image into fewer bits can be useful in reducing the storage requirements in image archival systems, or in decreasing the bandwidth for image transmission for applications such as teleconferencing and HDTV. Although some applications (e.g. medical imaging) require lossless compression, image compression usually introduces some loss in the original image. Another issue is the speed of compression and/or decompression, especially in real-time applications, In this paper the authors use a learning random neural network to achieve fast lossy image compression for gray level images. >
---
paper_title: Random Neural Networks with Negative and Positive Signals and Product Form Solution

paper_content:

We introduce a new class of random neural networks in which signals are either negative or positive. A positive signal arriving at a neuron increases its total signal count or potential by one; a negative signal reduces it by one if the potential is positive, and has no effect if it is zero. When its potential is positive, a neuron fires, sending positive or negative signals at random intervals to neurons or to the outside. Positive signals represent excitatory signals and negative signals represent inhibition. We show that this model, with exponential signal emission intervals, Poisson external signal arrivals, and Markovian signal movements between neurons, has a product form leading to simple analytical expressions for the system state.
---
paper_title: Video compression with random neural networks

paper_content:

We summarize a novel neural network technique for video compression, using a "point-process" type neural network model we have developed, which is closer to biophysical reality and is mathematically much more tractable than standard models. Our algorithm uses an adaptive approach based upon the users' desired video quality Q, and achieves compression ratios of up to 500:1 for moving gray-scale images, based on a combination of motion detection, compression and temporal subsampling of frames. This leads to a compression ratio of over 1000:1 for full-color video sequences with the addition of the standard 4:1:1 spatial subsampling ratios in the chrominance images. The signal-to-noise-ratio obtained varies with the compression level and ranges from 29 dB to over 34 dB. Our method is computationally fast so that compression and decompression could possibly be performed in real-time software.
---
paper_title: Video compression with wavelets and random neural network approximations

paper_content:

Modern video encoding techniques generate variable bit rates, because they take advantage of different rates of motion in scenes, in addition to using lossy compression within individual frames. We have introduced a novel method for video compression based on temporal subsampling of video frames, and for video frame reconstruction using neural network based function approximations. In this paper we describe another method using wavelets for still image compression of frames, and function approximations for the reconstruction of subsampled frames. We evaluated the performance of the method in terms of observed traffic characteristics for the resulting compressed and subsampled frames, and in terms of quality versus compression ratio curves with real video image sequences. Comparisons are presented with other standard methods.
---
paper_title: Joint Autoregressive and Hierarchical Priors for Learned Image Compression

paper_content:

Recent models for learned image compression are based on autoencoders, learning approximately invertible mappings from pixels to a quantized latent representation. These are combined with an entropy model, a prior on the latent representation that can be used with standard arithmetic coding algorithms to yield a compressed bitstream. Recently, hierarchical entropy models have been introduced as a way to exploit more structure in the latents than simple fully factorized priors, improving compression performance while maintaining end-to-end optimization. Inspired by the success of autoregressive priors in probabilistic generative models, we examine autoregressive, hierarchical, as well as combined priors as alternatives, weighing their costs and benefits in the context of image compression. While it is well known that autoregressive models come with a significant computational penalty, we find that in terms of compression performance, autoregressive and hierarchical priors are complementary and, together, exploit the probabilistic structure in the latents better than all previous learned models. The combined model yields state-of-the-art rate--distortion performance, providing a 15.8% average reduction in file size over the previous state-of-the-art method based on deep learning, which corresponds to a 59.8% size reduction over JPEG, more than 35% reduction compared to WebP and JPEG2000, and bitstreams 8.4% smaller than BPG, the current state-of-the-art image codec. To the best of our knowledge, our model is the first learning-based method to outperform BPG on both PSNR and MS-SSIM distortion metrics.
---
paper_title: Lossless Image Compression Using Reversible Integer Wavelet Transforms and Convolutional Neural Networks

paper_content:

In this work we introduce a lossless compression framework which incorporates convolutional neural networks (CNN) for wavelet subband prediction. A CNN is trained to predict detail coefficients from corresponding approximation coefficients, prediction error is then coded in place of wavelet coefficients. At decompression an identical CNN is used to reproduce the prediction and combine with the decoded residuals for perfect reconstruction of wavelet subbands
---
paper_title: Variational image compression with a scale hyperprior

paper_content:

We describe an end-to-end trainable model for image compression based on variational autoencoders. The model incorporates a hyperprior to effectively capture spatial dependencies in the latent representation. This hyperprior relates to side information, a concept universal to virtually all modern image codecs, but largely unexplored in image compression using artificial neural networks (ANNs). Unlike existing autoencoder compression methods, our model trains a complex prior jointly with the underlying autoencoder. We demonstrate that this model leads to state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and yields rate-distortion performance surpassing published ANN-based methods when evaluated using a more traditional metric based on squared error (PSNR). Furthermore, we provide a qualitative comparison of models trained for different distortion metrics.
---
paper_title: Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

paper_content:

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.
---
paper_title: Deep learning

paper_content:

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.
---
paper_title: Lossy Image Compression with Compressive Autoencoders

paper_content:

We propose a new approach to the problem of optimizing autoencoders for lossy image compression. New media formats, changing hardware technology, as well as diverse requirements and content types create a need for compression algorithms which are more flexible than existing codecs. Autoencoders have the potential to address this need, but are difficult to optimize directly due to the inherent non-differentiabilty of the compression loss. We here show that minimal changes to the loss are sufficient to train deep autoencoders competitive with JPEG 2000 and outperforming recently proposed approaches based on RNNs. Our network is furthermore computationally efficient thanks to a sub-pixel architecture, which makes it suitable for high-resolution images. This is in contrast to previous work on autoencoders for compression using coarser approximations, shallower architectures, computationally expensive methods, or focusing on small images.
---
paper_title: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

paper_content:

In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM.
---
paper_title: Long Short-Term Memory

paper_content:

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.
---
paper_title: Full Resolution Image Compression with Recurrent Neural Networks

paper_content:

This paper presents a set of full-resolution lossy image compression methods based on neural networks. Each of the architectures we describe can provide variable compression rates during deployment without requiring retraining of the network: each network need only be trained once. All of our architectures consist of a recurrent neural network (RNN)-based encoder and decoder, a binarizer, and a neural network for entropy coding. We compare RNN types (LSTM, associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study "one-shot" versus additive reconstruction architectures and introduce a new scaled-additive framework. We compare to previous work, showing improvements of 4.3%-8.8% AUC (area under the rate-distortion curve), depending on the perceptual metric used. As far as we know, this is the first neural network architecture that is able to outperform JPEG at image compression across most bitrates on the rate-distortion curve on the Kodak dataset images, with and without the aid of entropy coding.
---
paper_title: Spatially adaptive image compression using a tiled deep network

paper_content:

Deep neural networks represent a powerful class of function approximators that can learn to compress and reconstruct images. Existing image compression algorithms based on neural networks learn quantized representations with a constant spatial bit rate across each image. While entropy coding introduces some spatial variation, traditional codecs have benefited significantly by explicitly adapting the bit rate based on local image complexity and visual saliency. This paper introduces an algorithm that combines deep neural networks with quality-sensitive bit rate adaptation using a tiled network. We demonstrate the importance of spatial context prediction and show improved quantitative (PSNR) and qualitative (subjective rater assessment) results compared to a non-adaptive baseline and a recently published image compression model based on fully-convolutional neural networks.
---
paper_title: Real-Time Adaptive Image Compression

paper_content:

We present a machine learning-based approach to lossy image compression which outperforms all existing codecs, while running in real-time. ::: Our algorithm typically produces files 2.5 times smaller than JPEG and JPEG 2000, 2 times smaller than WebP, and 1.7 times smaller than BPG on datasets of generic images across all quality levels. At the same time, our codec is designed to be lightweight and deployable: for example, it can encode or decode the Kodak dataset in around 10ms per image on GPU. ::: Our architecture is an autoencoder featuring pyramidal analysis, an adaptive coding module, and regularization of the expected codelength. We also supplement our approach with adversarial training specialized towards use in a compression setting: this enables us to produce visually pleasing reconstructions for very low bitrates.
---
paper_title: DRAW: A Recurrent Neural Network For Image Generation

paper_content:

This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.
---
paper_title: Towards Conceptual Compression

paper_content:

We introduce a simple recurrent variational auto-encoder architecture that significantly improves image modeling. The system represents the state-of-the-art in latent variable models for both the ImageNet and Omniglot datasets. We show that it naturally separates global conceptual information from lower level details, thus addressing one of the fundamentally desired properties of unsupervised learning. Furthermore, the possibility of restricting ourselves to storing only global information about an image allows us to achieve high quality 'conceptual compression'.
---
paper_title: Light Field Image Compression Using Generative Adversarial Network-Based View Synthesis

paper_content:

Light field (LF) has become an attractive representation of immersive multimedia content for simultaneously capturing both the spatial and angular information of the light rays. In this paper, we present a LF image compression framework driven by a generative adversarial network (GAN)-based sub-aperture image (SAI) generation and a cascaded hierarchical coding structure. Specifically, we sparsely sample the SAIs in LF and propose the GAN of LF (LF-GAN) to generate the unsampled SAIs by analogy with adversarial learning conditioned on its surrounding contexts. In particular, the LF-GAN learns to interpret both the angular and spatial context of the LF structure and, meanwhile, generates intermediate hypothesis for the unsampled SAIs in a certain position. Subsequently, the sampled SAIs and the residues of the generated-unsampled SAIs are re-organized as pseudo-sequences and compressed by standard video codecs. Finally, the hierarchical coding structure is adopted for the sampled SAI to effectively remove the inter-view redundancies. During the training process of LF-GAN, the pixel-wise Euclidean loss and the adversarial loss are chosen as the optimization objective, such that sharp textures with less blurring in details can be produced. Extensive experimental results show that the proposed LF-GAN-based LF image compression framework outperforms the state-of-the-art learning-based LF image compression approach with on average 4.9% BD-rate reductions over multiple LF datasets.
---
paper_title: A Hybrid Neural Network for Chroma Intra Prediction

paper_content:

For chroma intra prediction, previous methods exemplified by the Linear Model method (LM) usually assume a linear correlation between the luma and chroma components in a coding block. This assumption is inaccurate for complex image content or large blocks, and restricts the prediction accuracy. In this paper, we propose a chroma intra prediction method by exploiting both spatial and cross-channel correlations using a hybrid neural network. Specifically, we utilize a convolutional neural network to extract features from the reconstructed luma samples of the current block, as well as utilize a fully connected network to extract features from the neighboring reconstructed luma and chroma samples. The extracted twofold features are then fused to predict the chroma samples–Cb and Cr simultaneously. The proposed chroma intra prediction method is integrated into HEVC. Preliminary results show that, compared with HEVC plus LM, the proposed method achieves on average 0.2%, 3.1% and 2.0% BD-rate reduction on Y, Cb and Cr components, respectively, under All-Intra configuration.
---
paper_title: A Dual-Network Based Super-Resolution for Compressed High Definition Video

paper_content:

Convolutional neural network (CNN) based super-resolution (SR) has achieved superior performance compared with traditional methods for uncompressed images/videos, but its performance degenerates dramatically for compressed content especially at low bit-rate scenario due to the mixture distortions during sampling and compressing. This is critical because images/videos are always compressed with degraded quality in practical scenarios. In this paper, we propose a novel dual-network structure to improve the CNN-based SR performance for compressed high definition video especially at low bit-rate. To alleviate the impact of compression, an enhancement network is proposed to remove the compression artifacts which is located ahead of the SR network. The two networks, enhancement network and SR network, are optimized stepwise for different tasks of compression artifact reduction and SR respectively. Moreover, an improved geometric self-ensemble strategy is proposed to further improve the SR performance. Extensive experimental results demonstrate that the dual-network scheme can significantly improve the quality of super-resolved images/videos compared with those reconstructed from single SR network for compressed content. It achieves around 31.5% bit-rate saving for 4 K video compression compared with HEVC when applying the proposed method in a SR-based video coding framework, which proves the potential of our method in practical scenarios, e.g., video coding and SR.
---
paper_title: Neural network based intra prediction for video coding

paper_content:

Today’s hybrid video coding systems typically perform an intra-picture prediction whereby blocks of samples are predicted from previously decoded samples of the same picture. For example, HEVC uses a set of angular prediction patterns to exploit directional sample correlations. In this paper, we propose new intra-picture prediction modes whose construction consists of two steps: First, a set of features is extracted from the decoded samples. Second, these features are used to select a predefined image pattern as the prediction signal. Since several intra prediction modes are proposed for each block-shape, a specific signalization scheme is also proposed. Our intra prediction modes lead to significant coding gains over state of the art video coding technologies.
---
paper_title: Toward a new video compression scheme using super-resolution

paper_content:

The term super-resolution is typically used in the literature to describe the process of obtaining a high resolution (HR) image or a sequence of HR images from a set of low resolution (LR) observations. This term has been applied primarily to spatial and temporal resolution enhancement. However, intentional pre-processing and downsampling can be applied during encoding and super-resolution techniques to upsample the image can be applied during decoding when video compression is the main objective. In this paper we consider the following three video compression models. The first one simply compresses the sequence using any of the available standard compression methods, the second one pre-processes (without downsampling) the image sequence before compression, so that post-processing (without upsampling) is applied to the compressed sequence. The third model includes downsampling in the pre-processing stage and the application of a super resolution technique during decoding. In this paper we describe these three models but concentrate ::: on the application of super-resolution techniques as a way to post-process and upsample a compressed video sequences. Experimental results are provided on a wide range of bitrates for two very important applications: format conversion between different platforms and scalable video coding.
---
paper_title: Efficient CTU-based intra frame coding for HEVC based on deep learning

paper_content:

To further improve the compression efficiency of HEVC intra frame coding, in this paper, a deep learning-based framework is proposed. Inspired by recently developed deep learning models for image super-resolution (SR), we propose to train a CNN (convolutional neural network) model to precisely predict the residual information of each CTU (coding tree unit) at the HEVC encoder. As a result, better CTU reconstruction and better prediction for the compression of subsequent CTUs can be achieved. To reduce computational complexity, different from current CNN-based SR works, we propose to skip the non-linear mapping layer, and incorporate the residual learning to obtain better predicted residual for CTU encoding. Experimental results have shown that the proposed method achieves 3.2% bitrate reduction in average BDBR (Bjentegaard delta bit rate) with only 37% encoding complexity increased.
---
paper_title: Enhanced Intra Prediction with Recurrent Neural Network in Video Coding

paper_content:

Intra prediction is one of the important parts in video/image codec. With intra prediction mechanism, spatial redundancy can be largely removed for further bit saving. However, current state-of-the-art intra prediction method does not produce satisfactory prediction result due to its limits in reference samples and modeling ability. To enhance the intra prediction in HEVC, in this paper, a deep neural network featuring spatial RNN, which models the spatial dependency of pixels as sequential dynamics, is proposed to generate better prediction signals. Experimental results show improvement in BD-Rate for the proposed method compared with the original HEVC prediction scheme.
---
paper_title: Learning a Convolutional Neural Network for Image Compact-Resolution

paper_content:

We study the dual problem of image super-resolution (SR), which we term image compact-resolution (CR). Opposite to image SR that hallucinates a visually plausible high-resolution image given a low-resolution input, image CR provides a low-resolution version of a high-resolution image, such that the low-resolution version is both visually pleasing and as informative as possible compared to the high-resolution image. We propose a convolutional neural network (CNN) for image CR, namely, CNN-CR, inspired by the great success of CNN for image SR. Specifically, we translate the requirements of image CR into operable optimization targets for training CNN-CR: the visual quality of the compact resolved image is ensured by constraining its difference from a naively downsampled version and the information loss of image CR is measured by upsampling/super-resolving the compact-resolved image and comparing that to the original image. Accordingly, CNN-CR can be trained either separately or jointly with a CNN for image SR. We explore different training strategies as well as different network structures for CNN-CR. Our experimental results show that the proposed CNN-CR clearly outperforms simple bicubic downsampling and achieves on average 2.25 dB improvement in terms of the reconstruction quality on a large collection of natural images. We further investigate two applications of image CR, i.e., low-bit-rate image compression and image retargeting. Experimental results show that the proposed CNN-CR helps achieve significant bits saving than High Efficiency Video Coding when applied to image compression and produce visually pleasing results when applied to image retargeting.
---
paper_title: Convolutional Neural Network-Based Block Up-Sampling for Intra Frame Coding

paper_content:

Inspired by the recent advances of image super-resolution using convolutional neural network (CNN), we propose a CNN-based block up-sampling scheme for intra frame coding. A block can be down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution. Different from previous studies on down/up-sampling-based coding, the up-sampling methods in our scheme have been designed by training CNN instead of hand-crafted. We explore a new CNN structure for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning, making the network both compact and efficient. We also design different networks for the up-sampling of luma and chroma components, respectively, where the chroma up-sampling CNN utilizes the luma information to boost its performance. In addition, we design a two-stage up-sampling process, the first stage being within the block-by-block coding loop, and the second stage being performed on the entire frame, so as to refine block boundaries. We also empirically study how to set the coding parameters of down-sampled blocks for pursuing the frame-level rate-distortion optimization. Our proposed scheme is implemented into the high-efficiency video coding (HEVC) reference software, and a comprehensive set of experiments have been performed to evaluate our methods. Experimental results show that our scheme achieves significant bits saving compared with the HEVC anchor, especially at low bit rates, leading to on average 5.5% BD-rate reduction on common test sequences and on average 9.0% BD-rate reduction on ultrahigh definition test sequences.
---
paper_title: Convolutional Neural Network-Based Block Up-Sampling for HEVC

paper_content:

Recently, convolutional neural network (CNN)-based methods have achieved remarkable progress in image and video super-resolution, which inspires research on down-/up-sampling-based image and video coding using CNN. Instead of hand-crafted filters for up-sampling, trained CNN models are believed to be more capable of improving image quality, thus leading to coding gain. However, previous studies either concentrated on intra-frame coding or performed down- and up-sampling of entire frame. In this paper, we introduce block-level down- and up-sampling into inter-frame coding with the help of CNN. Specifically, each block in the P or B frame can either be compressed at the original resolution or down-sampled and compressed at low resolution and then, up-sampled by the trained CNN models. Such block-level adaptivity is flexible to cope with the spatially variant texture and motion characteristics. We further investigate how to enhance the capability of CNN-based up-sampling by utilizing reference frames and study how to train the CNN models by using encoded video sequences. We implement the proposed scheme onto the high efficiency video coding (HEVC) reference software and perform a comprehensive set of experiments to evaluate our methods. The experimental results show that our scheme achieves superior performance to the HEVC anchor, especially at low bit rates, leading to an average 3.8%, 2.6%, and 3.5% BD-rate reduction on the HEVC common test sequences under random-access, low-delay B, and low-delay P configurations, respectively. When tested on high-definition and ultrahigh-definition sequences, the average BD-rate exceeds 5%.
---
paper_title: Fully Connected Network-Based Intra Prediction for Image Coding

paper_content:

This paper proposes a deep learning method for intra prediction. Different from traditional methods utilizing some fixed rules, we propose using a fully connected network to learn an end-to-end mapping from neighboring reconstructed pixels to the current block. In the proposed method, the network is fed by multiple reference lines. Compared with traditional single line-based methods, more contextual information of the current block is utilized. For this reason, the proposed network has the potential to generate better prediction. In addition, the proposed network has good generalization ability on different bitrate settings. The model trained from a specified bitrate setting also works well on other bitrate settings. Experimental results demonstrate the effectiveness of the proposed method. When compared with high efficiency video coding reference software HM-16.9, our network can achieve an average of 3.4% bitrate saving. In particular, the average result of 4K sequences is 4.5% bitrate saving, where the maximum one is 7.4%.
---
paper_title: Down-Sampling Based Video Coding Using Super-Resolution Technique

paper_content:

It has been reported that oversampling a still image before compression does not guarantee a good image quality. Similarly, down-sampling before video compression in low bit rate video coding may alleviate the blocking effect and improve peak signal-to-noise ratio of the decoded frames. When the number of discrete cosine transform coefficients is reduced in such a down-sampling based coding (DBC), the bit budget of each coefficient will increase, thus reduce the quantization error. A DBC video coding scheme is proposed in this paper, where a super-resolution technique is employed to restore the down-sampled frames to their original resolutions. The performance improvement of the proposed DBC scheme is analyzed at low bit rates, and verified by experiments.
---
paper_title: Video Frame Interpolation via Adaptive Separable Convolution

paper_content:

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.
---
paper_title: Adaptive Interpolation Filter for H.264/AVC

paper_content:

In order to reduce the bit-rate of video signals, current coding standards apply hybrid coding with motion-compensated prediction and transform coding of the prediction error. In former publications, it has been shown that aliasing components contained in an image signal, as well as motion blur are limiting the prediction efficiency obtained by motion compensation. In this paper, we show that the analytical calculation of an optimal interpolation filter at particular constraints is possible, resulting in total coding improvements of 20% at broadcast quality compared to the H.264/AVC High Profile. Furthermore, the spatial adaptation to local image characteristics enables further improvements of 0.15 dB for CIF sequences compared to globally adaptive filter or up to 0.6 dB, compared to the standard H.264/AVC. Additionally, we show that the presented approach is generally applicable, i.e., also motion blur can be exactly compensated, if particular constraints are fulfilled.
---
paper_title: Convolutional Neural Network-Based Motion Compensation Refinement for Video Coding

paper_content:

Inspired by the great success of convolutional neural network (CNN) in computer vision, we propose a CNN-based method to refine the motion compensation in video coding. First, we study a simple CNN-based motion compensation refinement (CNNMCR) scheme, where we train a CNN to refine the motion compensated prediction directly. Second, we consider to exploit the contextual information for the refinement, and propose a more powerful CNNMCR scheme, where the CNN utilizes not only the motion compensated prediction, but also the neighboring reconstructed region to refine the prediction. We integrate the simple CNNMCR and the CNNMCR schemes into the High Efficiency Video Coding (HEVC) framework. Experimental results show that both schemes achieve better compression performance than the HEVC anchor, leading to on average 1.8% and 2.3% BD-rate reduction, respectively, under low-delay P configuration. Furthermore, the combination of our proposed CNNMCR and the overlapped block motion compensation (OBMC) technique provides as high as 5.2% BD-rate reduction.
---
paper_title: Learning a convolutional neural network for fractional interpolation in HEVC inter coding

paper_content:

Motion compensated prediction (MCP) is an effective technology for video coding to improve compression efficiency. Fractional sample precision prediction is utilized in HEVC to further remove temporal redundancy, and finite impulse response (FIR) filters designed using decomposition of the discrete cosine transform are applied to generate samples that do not fall on the integer positions. However, the coefficients of these DCT-based interpolation filters are fixed, which may not be able to adapt to varied video content. Inspired by the remarkable success of convolutional neural network (CNN) in the single image super-resolution task, we propose to learn a convolutional neural network for fractional interpolation in HEVC inter prediction. Compared with super-resolution, there is one big difference in fractional interpolation — fractional interpolation needs to maintain samples at integer positions while super-resolution generates a whole high-resolution image. Another difference is no real ground truth is available in fractional interpolation process. To overcome these two challenges, we introduce a constraint strategy to the training phase of the original super-resolution network as well as a specially designed preprocessing step which reuses the DCTIF interpolation process. Unlike other previous work, our proposed approach simultaneously generating the fractional positions from one network and experimental results show our proposed approach achieves 0.45% BD-Rate reduction under the low-delay-P configuration on average.
---
paper_title: A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding

paper_content:

Lossy image and video compression algorithms yield visually annoying artifacts including blocking, blurring, and ringing, especially at low bit-rates. To reduce these artifacts, post-processing techniques have been extensively studied. Recently, inspired by the great success of convolutional neural network (CNN) in computer vision, some researches were performed on adopting CNN in post-processing, mostly for JPEG compressed images. In this paper, we present a CNN-based post-processing algorithm for High Efficiency Video Coding (HEVC), the state-of-the-art video coding standard. We redesign a Variable-filter-size Residue-learning CNN (VRCNN) to improve the performance and to accelerate network training. Experimental results show that using our VRCNN as post-processing leads to on average 4.6% bit-rate reduction compared to HEVC baseline. The VRCNN outperforms previously studied networks in achieving higher bit-rate reduction, lower memory cost, and multiplied computational speedup.
---
paper_title: Convolutional Neural Network-Based Fractional-Pixel Motion Compensation

paper_content:

Fractional-pixel motion compensation (MC) improves the efficiency of inter prediction and has been utilized extensively in video coding standards. The traditional methods of fractional-pixel MC usually follow the approach of interpolation, i.e., they adopt different kinds of filters, either fixed or adaptive, to interpolate fractional-pixel values from integer-pixel values in a reference picture. Different from the interpolation approach, in this paper, we formulate the fractional-pixel MC as an inter-picture regression problem, which is to predict the pixel values of the current to-be-coded picture from the integer-pixel values of a reference picture, given a fractional-pixel motion vector that relates the two pictures. We then propose to adopt convolutional neural network (CNN) models to approach the regression problem, inspired by the recent advances of CNN. Accordingly, we propose fractional-pixel reference generation CNN (FRCNN) for both uni-directional and bi-directional MC in video coding. We further investigate how to train FRCNN by using encoded video sequences, and empirically study the effect of different training data and different CNN structures. Moreover, we propose to integrate FRCNN into the high efficiency video coding (HEVC) scheme, and perform a comprehensive set of experiments to evaluate the effectiveness of FRCNN. The experimental results show that our proposed FRCNN achieves on average 3.9%, 2.7%, and 1.3% bits saving compared with HEVC, under low-delay P, low-delay B, and random-access configurations, respectively.
---
paper_title: A comparison of fractional-pel interpolation filters in HEVC and H.264/AVC

paper_content:

The fractional-pel interpolation filter adopted in H.264/AVC improves motion compensation greatly. Recently, a new DCT-based fractional-pel interpolation filter is adopted in the oncoming standard HEVC. We are interested in the differences between these two types of fractional-pel interpolation filters. In this paper we describe the derivations of fractional-pel interpolation filters in HEVC and H.264/AVC in detail, and compare them on properties of frequency responses. We find that the half-pel interpolation filters in HEVC and H.264/AVC are very similar, but the low-pass properties of quarter-pel interpolation filters in HEVC are much better than those in H.264/AVC. Experimental results validate this phenomenon, the fractional-pel interpolation in H.264/AVC tends to increase BD-rates by more than 10% compared with that in HEVC, and this performance loss mainly comes from quarter-pel interpolation filters. On the other hand, the complexity of fractional-pel interpolation filtering in HEVC is greatly increased than that in H.264/AVC.
---
paper_title: Enhanced Ctu-Level Inter Prediction with Deep Frame Rate Up-Conversion for High Efficiency Video Coding

paper_content:

Inter prediction serves as the foundation of prediction based hybrid video coding framework. The state-of-the-art video coding standards employ the reconstructed frames as the references, and the motion vectors which convey the relative position shift between the current block and the prediction block are explicitly signalled in the bitstream. In this paper, we propose a high efficient inter prediction scheme by introducing a new methodology based on virtual reference frame, which is effectively generated with the deep neural network such that the motion data does not need to be explicitly signalled. In particular, the high quality virtual reference frame is generated with the deep learning based frame rate up-conversion (FRUC) algorithm from two reconstructed bi-prediction frames. Subsequently, a novel CTU level coding mode termed as direct virtual reference frame (DVRF) mode, is proposed to adaptively compensate for the current to-be-coded block in the sense of rate-distortion optimization (RDO). The proposed scheme is integrated into the HM-16.6 software, and experimental results demonstrate significant superiority of the proposed method, which provides more than 3% coding gains on average for HEVC test sequences.
---
paper_title: Convolutional Neural Network-Based Invertible Half-Pixel Interpolation Filter for Video Coding

paper_content:

Fractional-pixel interpolation has been widely used in the modern video coding standards to improve the accuracy of motion compensated prediction. Traditional interpolation filters are designed based on the signal processing theory. However, video signal is non-stationary, making the traditional methods less effective. In this paper, we reveal that the interpolation filter can not only generate the fractional pixels from the integer pixels, but also reconstruct the integer pixels from the fractional ones. This property is called invertibility. Inspired by the invertibility of fractional-pixel interpolation, we propose an end-to-end scheme based on convolutional neural network (CNN) to derive the invertible interpolation filter, termed CNNInvIF. CNNlnvIF does not need the “ground-truth” of fractional pixels for training. Experimental results show that the proposed CNNInvIF can achieve up to 4.6% and on average 2.2% BD-rate reduction than HEVC under the low-delay P configuration.
---
paper_title: Enhanced Bi-Prediction With Convolutional Neural Network for High-Efficiency Video Coding

paper_content:

In this paper, we propose an enhanced bi-prediction scheme based on the convolutional neural network (CNN) to improve the rate-distortion performance in video compression. In contrast to the traditional bi-prediction strategy which computes the linear superposition as the predictive signals with pixel-to-pixel correspondence, the proposed scheme employs CNN to directly infer the predictive signals in a data-driven manner. As such, the predicted blocks are fused in a nonlinear fashion to improve the coding performance. Moreover, the patch-to-patch inference strategy with CNN also improves the prediction accuracy since the patch-level information for the prediction of each individual pixel can be exploited. The proposed enhanced bi-prediction scheme is further incorporated into the high-efficiency video coding standard, and the experimental results exhibit a significant performance improvement under different coding configurations.
---
paper_title: Enhanced Motion-Compensated Video Coding With Deep Virtual Reference Frame Generation

paper_content:

In this paper, we propose an efficient inter prediction scheme by introducing the deep virtual reference frame (VRF), which serves better reference in the temporal redundancy removal process of video coding. In particular, the high quality VRF is generated with the deep learning-based frame rate up conversion (FRUC) algorithm from two reconstructed bi-directional frames, which is subsequently incorporated into the reference list serving as the high quality reference. Moreover, to alleviate the compression artifacts of VRF, we develop a convolutional neural network (CNN)-based enhancement model to further improve its quality. To facilitate better utilization of the VRF, a CTU level coding mode termed as direct virtual reference frame (DVRF) is devised, which achieves better trade-off between compression performance and complexity. The proposed scheme is integrated into HM-16.6 and JEM-7.1 software platforms, and the simulation results under random access (RA) configuration demonstrate significant superiority of the proposed method. When adding VRF to RPS, more than 6% average BD-rate gain is achieved for HEVC test sequences on HM-16.6, and 0.8% BD-rate gain is observed based on JEM-7.1 software. Regarding the DVRF mode, 3.6% bitrate saving is achieved on HM-16.6 with the computational complexity effectively reduced.
---
paper_title: Gradient-Based Learning Applied to Document Recognition

paper_content:

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.
---
paper_title: Image quality assessment: from error visibility to structural similarity

paper_content:

Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.
---
paper_title: A perceptual quantization strategy for HEVC based on a convolutional neural network trained on natural images

paper_content:

Fast prediction models of local distortion visibility and local quality can potentially make modern spatiotemporally adaptive coding schemes feasible for real-time applications. In this paper, a fast convolutional-neural- network based quantization strategy for HEVC is proposed. Local artifact visibility is predicted via a network trained on data derived from our improved contrast gain control model. The contrast gain control model was trained on our recent database of local distortion visibility in natural scenes [Alam et al. JOV 2014]. Further- more, a structural facilitation model was proposed to capture effects of recognizable structures on distortion visibility via the contrast gain control model. Our results provide on average 11% improvements in compression efficiency for spatial luma channel of HEVC while requiring almost one hundredth of the computational time of an equivalent gain control model. Our work opens the doors for similar techniques which may work for different forthcoming compression standards.
---
paper_title: Neural network-based arithmetic coding of intra prediction modes in HEVC

paper_content:

In both H.264 and HEVC, context-adaptive binary arithmetic coding (CABAC) is adopted as the entropy coding method. CABAC relies on manually designed binarization processes as well as handcrafted context models, which may restrict the compression efficiency. In this paper, we propose an arithmetic coding strategy by training neural networks, and make preliminary studies on coding of the intra prediction modes in HEVC. Instead of binarization, we propose to directly estimate the probability distribution of the 35 intra prediction modes with the adoption of a multi-level arithmetic codec. Instead of handcrafted context models, we utilize convolutional neural network (CNN) to perform the probability estimation. Simulation results show that our proposed arithmetic coding leads to as high as 9.9% bits saving compared with CABAC.
---
paper_title: CNN-based transform index prediction in multiple transforms framework to assist entropy coding

paper_content:

Recent work in video compression has shown that using multiple 2D transforms instead of a single transform in order to de-correlate residuals provides better compression efficiency. These transforms are tested competitively inside a video encoder and the optimal transform is selected based on the Rate Distortion Optimization (RDO) cost. However, one needs to encode a syntax to indicate the chosen transform per residual block to the decoder for successful reconstruction of the pixels. Conventionally, the transform index is binarized using fixed length coding and a CABAC context model is attached to it. In this work, we provide a novel method that utilizes Convolutional Neural Network to predict the chosen transform index from the quantized coefficient block. The prediction probabilities are used to binarize the index by employing a variable length coding instead of a fixed length coding. Results show that by employing this modified transform index coding scheme inside HEVC, one can achieve up to 0.59% BD-rate gain.
---
paper_title: Convolutional Neural Network-Based Synthesized View Quality Enhancement for 3D Video Coding

paper_content:

The quality of synthesized view plays an important role in the three dimensional (3D) video system. In this paper, to further improve the coding efficiency, a convolutional neural network (CNN) based synthesized view quality enhancement method for 3D High Efficiency Video Coding (HEVC) is proposed. Firstly, the distortion elimination in synthesized view is formulated as an image restoration task with the aim to reconstruct the latent distortion free synthesized image. Secondly, the learned CNN models are incorporated into 3D HEVC codec to improve the view synthesis performance for both view synthesis optimization (VSO) and the final synthesized view, where the geometric and compression distortions are considered according to the specific characteristics of synthesized view. Thirdly, a new Lagrange multiplier in the rate-distortion (RD) cost function is derived to adapt the CNN based VSO process to embrace a better 3D video coding performance. Extensive experimental results show that the proposed scheme can efficiently eliminate the artifacts in the synthesized image, and reduce 25.9% and 11.7% bit rate in terms of peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) index, which significantly outperforms the state-of-theart methods.
---
paper_title: Content-Aware Convolutional Neural Network for In-Loop Filtering in High Efficiency Video Coding

paper_content:

Recently, convolutional neural network (CNN) has attracted tremendous attention and has achieved great success in many image processing tasks. In this paper, we focus on CNN technology combined with image restoration to facilitate video coding performance and propose the content-aware CNN based in-loop filtering for high-efficiency video coding (HEVC). In particular, we quantitatively analyze the structure of the proposed CNN model from multiple dimensions to make the model interpretable and optimal for CNN-based loop filtering. More specifically, each coding tree unit (CTU) is treated as an independent region for processing, such that the proposed content-aware multimodel filtering mechanism is realized by the restoration of different regions with different CNN models under the guidance of the discriminative network. To adapt the image content, the discriminative neural network is learned to analyze the content characteristics of each region for the adaptive selection of the deep learning model. The CTU level control is also enabled in the sense of rate-distortion optimization. To learn the CNN model, an iterative training method is proposed by simultaneously labeling filter categories at the CTU level and fine-tuning the CNN model parameters. The CNN based in-loop filter is implemented after sample adaptive offset in HEVC, and extensive experiments show that the proposed approach significantly improves the coding performance and achieves up to 10.0% bit-rate reduction. On average, 4.1%, 6.0%, 4.7%, and 6.0% bit-rate reduction can be obtained under all intra, low delay, low delay P, and random access configurations, respectively.
---
paper_title: CAS-CNN: A Deep Convolutional Neural Network for Image Compression Artifact Suppression

paper_content:

Lossy image compression algorithms are pervasively used to reduce the size of images transmitted over the web and recorded on data storage media. However, we pay for their high compression rate with visual artifacts degrading the user experience. Deep convolutional neural networks have become a widespread tool to address high-level computer vision tasks very successfully. Recently, they have found their way into the areas of low-level computer vision and image processing to solve regression problems mostly with relatively shallow networks. We present a novel 12-layer deep convolutional network for image compression artifact suppression with hierarchical skip connections and a multi-scale loss function. We achieve a boost of up to 1.79 dB in PSNR over ordinary JPEG and an improvement of up to 0.36 dB over the best previous ConvNet result. We show that a network trained for a specific quality factor (QF) is resilient to the QF used to compress the input image — a single network trained for QF 60 provides a PSNR gain of more than 1.5 dB over the wide QF range from 40 to 76.
---
paper_title: Just-Noticeable Difference-Based Perceptual Optimization for JPEG Compression

paper_content:

The Quantization table in JPEG, which specifies the quantization scale for each discrete cosine transform (DCT) coefficient, plays an important role in image codec optimization. However, the generic quantization table design that is based on the characteristics of human visual system (HVS) cannot adapt to the variations of image content. In this letter, we propose a just-noticeable difference (JND) based quantization table derivation method for JPEG by optimizing the rate-distortion costs for all the frequency bands. To achieve better perceptual quality, the DCT domain JND-based distortion metric is utilized to model the stair distortion perceived by HVS. The rate-distortion cost for each band is derived by estimating the rate with the first-order entropy of quantized coefficients. Subsequently, the optimal quantization table is obtained by minimizing the total rate-distortion costs of all the bands. Extensive experimental results show that the quantization table generated by the proposed method achieves significant bit-rate savings compared with JPEG recommended quantization table and specifically developed quantization tables in terms of both objective and subjective evaluations.
---
paper_title: CNN-based in-loop filtering for coding efficiency improvement

paper_content:

A recent video coding standard, called High Efficiency Video Coding (HEVC), adopts two in-loop filters for coding efficiency improvement where the in-loop filtering is done by a de-blocking filter (DF) followed by sample adaptive offset (SAO) filtering. The DF helps improve both coding efficiency and subjective quality without signaling any bit to decoder sides while SAO filtering corrects the quantization errors by sending offset values to decoders. In this paper, we first present a new in-loop filtering technique using convolutional neural networks (CNN), called IFCNN, for coding efficiency and subjective visual quality improvement. The IFCNN does not require signaling bits by using the same trained weights in both encoders and decoder. The proposed IFCNN is trained in two different QP ranges: QR1 from QP = 20 to QP = 29; and QR2 from QP = 30 to QP = 39. In testing, the IFCNN trained in QR1 is applied for the encoding/decoding with QP values less than 30 while the IFCNN trained in QR2 is applied for the case of QP values greater than 29. The experiment results show that the proposed IFCNN outperforms the HEVC reference mode (HM) with average 1.9%–2.8% gain in BD-rate for Low Delay configuration, and average 1.6%–2.6% gain in BD-rate for Random Access configuration with IDR period 16.
---
paper_title: Nonlocal In-Loop Filter: The Way Toward Next-Generation Video Coding?

paper_content:

In-loop filtering has emerged as an essential coding tool since H.264/AVC, due to its delicate design, which reduces different kinds of compression artifacts. However, existing in-loop filters rely only on local image correlations, largely ignoring nonlocal similarities. In this article, the authors explore the design philosophy of in-loop filters and discuss their vision for the future of in-loop filter research by examining the potential of nonlocal similarities. Specifically, the group-based sparse representation, which jointly exploits an image's local and nonlocal self-similarities, lays a novel and meaningful groundwork for in-loop filter design. Hard- and soft-thresholding filtering operations are applied to derive the sparse parameters that are appropriate for compression artifact reduction. Experimental results show that this in-loop filter design can significantly improve the compression performance of the High Efficiency Video Coding (HEVC) standard, leading us in a new direction for improving compression efficiency.
---
paper_title: An efficient deep convolutional neural networks model for compressed image deblocking

paper_content:

Convolutional neural networks (CNNs) have been widely used in image processing community. Image deblocking is a post-processing strategy, which aims to reduce the visually annoying blocking artifacts that are caused by block-based transform coding at low bit rates. In recent years, CNNs based methods have been proposed to solve this classic image processing problem. In this paper, we present an efficient deep C-NNs model for image deblocking. Our model can well alleviate the conflict between bit reduction and quality preservation by taking local small patches into consideration. Our trained model can be used to deblock lossy compressed images with different quality factors. The proposed model can be easily integrated into the existing codecs as a post-processing procedure without changing the codec. Experimental results verify that our proposed model outperforms the state-of-the-art methods in both the objective quality and the perceptual quality.
---
paper_title: Low-Rank-Based Nonlocal Adaptive Loop Filter for High-Efficiency Video Compression

paper_content:

In video coding, the in-loop filtering has emerged as a key module due to its significant improvement on compression performance since H.264/Advanced Video Coding. Existing incorporated in-loop filters in video coding standards mainly take advantage of the local smoothness prior model used for images. In this paper, we propose a novel adaptive loop filter utilizing image nonlocal prior knowledge by imposing the low-rank constraint on similar image patches for compression noise reduction. In the filtering process, the reconstructed frame is first divided into image patch groups according to image patch similarity. The proposed in-loop filtering is formulated as an optimization problem with low-rank constraint for every group of image patches independently. It can be efficiently solved by soft-thresholding singular values of the matrix composed of image patches in the same group. To adapt the properties of the input sequences and bit budget, an adaptive threshold derivation model is established for every group of image patches according to the characteristics of compressed image patches, quantization parameters, and coding modes. Moreover, frame-level and largest coding unit-level control flags are signaled to further improve the adaptability from the sense of rate-distortion optimization. The performance of the proposed in-loop filter is analyzed when it collaborates with the existing in-loop filters in High Efficiency Video Coding. Extensive experimental results show that our proposed in-loop filter can further improve the performance of state-of-the-art video coding standard significantly, with up to 16% bit-rate savings.
---
paper_title: S-Net: a scalable convolutional neural network for JPEG compression artifact reduction

paper_content:

Recent studies have used deep residual convolutional neural networks (CNNs) for JPEG compression artifact reduction. This study proposes a scalable CNN called S-Net. Our approach effectively adjusts the network scale dynamically in a multitask system for real-time operation with little performance loss. It offers a simple and direct technique to evaluate the performance gains obtained with increasing network depth, and it is helpful for removing redundant network layers to maximize the network efficiency. We implement our architecture using the Keras framework with the TensorFlow backend on an NVIDIA K80 GPU server. We train our models on the DIV2K dataset and evaluate their performance on public benchmark datasets. To validate the generality and universality of the proposed method, we created and utilized a new dataset, called WIN143, for over-processed images evaluation. Experimental results indicate that our proposed approach outperforms other CNN-based methods and achieves state-of-the-art performance.
---
paper_title: Compression Artifacts Reduction by a Deep Convolutional Network

paper_content:

Lossy compression introduces complex compression artifacts, particularly the blocking artifacts, ringing effects and blurring. Existing algorithms either focus on removing blocking artifacts and produce blurred output, or restores sharpened images that are accompanied with ringing effects. Inspired by the deep convolutional networks (DCN) on super-resolution, we formulate a compact and efficient network for seamless attenuation of different compression artifacts. We also demonstrate that a deeper model can be effectively trained with the features learned in a shallow network. Following a similar"easy to hard"idea, we systematically investigate several practical transfer settings and show the effectiveness of transfer learning in low-level vision problems. Our method shows superior performance than the state-of-the-arts both on the benchmark datasets and the real-world use case (i.e. Twitter). In addition, we show that our method can be applied as pre-processing to facilitate other low-level vision routines when they take compressed images as input.
---
paper_title: Spatial-Temporal Residue Network Based In-Loop Filter for Video Coding

paper_content:

Deep learning has demonstrated tremendous break through in the area of image/video processing. In this paper, a spatial-temporal residue network (STResNet) based in-loop filter is proposed to suppress visual artifacts such as blocking, ringing in video coding. Specifically, the spatial and temporal information is jointly exploited by taking both current block and co-located block in reference frame into consideration during the processing of in-loop filter. The architecture of STResNet only consists of four convolution layers which shows hospitality to memory and coding complexity. Moreover, to fully adapt the input content and improve the performance of the proposed in-loop filter, coding tree unit (CTU) level control flag is applied in the sense of rate-distortion optimization. Extensive experimental results show that our scheme provides up to 5.1% bit-rate reduction compared to the state-of-the-art video coding standard.
---
paper_title: Multi-Frame Quality Enhancement for Compressed Video

paper_content:

The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, ignoring the similarity between consecutive frames. In this paper, we investigate that heavy quality fluctuation exists across compressed video frames, and thus low quality frames can be enhanced using the neighboring high quality frames, seen as Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as a first attempt in this direction. In our approach, we firstly develop a Support Vector Machine (SVM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are as the input. The MF-CNN compensates motion between the non-PQF and PQFs through the Motion Compensation subnet (MC-subnet). Subsequently, the Quality Enhancement subnet (QE-subnet) reduces compression artifacts of the non-PQF with the help of its nearest PQFs. Finally, the experiments validate the effectiveness and generality of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video. The code of our MFQE approach is available at https://github.com/ryangBUAA/MFQE.git.
---
paper_title: Learning a Deep Convolutional Network for Image Super-Resolution

paper_content:

We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) [15] that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage.
---
paper_title: HEVC Deblocking Filter

paper_content:

This paper describes the in-loop deblocking filter used in the upcoming High Efficiency Video Coding (HEVC) standard to reduce visible artifacts at block boundaries. The deblocking filter performs detection of the artifacts at the coded block boundaries and attenuates them by applying a selected filter. Compared to the H.264/AVC deblocking filter, the HEVC deblocking filter has lower computational complexity and better parallel processing capabilities while still achieving significant reduction of the visual artifacts.
---
paper_title: H.263+: Video Coding at Low Bit Rates

paper_content:

We discuss the ITU-T H.263+ (or H.263 Version 2) low-bit-rate video coding standard. We first describe, briefly, the H.263 standard including its optional modes. We then address the 12 new negotiable modes of H.263+. Next, we present experimental results for these modes, based on our public-domain implementation (http:ilspmg.ece.ubc.ca). Tradeoffs among compression performance, complexity, and memory requirements for the H.263+ optional modes are discussed. Finally, results for mode combinations are presented.
---
paper_title: A Practical Convolutional Neural Network as Loop Filter for Intra Frame

paper_content:

Loop filters are used in video coding to remove artifacts or improve performance. Recent advances in deploying convolutional neural network (CNN) to replace traditional loop filters show large gains but with problems for practical application. First, different model is used for frames encoded with different quantization parameter (QP), respectively. It is expensive for hardware. Second, float points operation in CNN leads to inconsistency between encoding and decoding across different platforms. Third, redundancy within CNN model consumes precious computational resources. This paper proposes a CNN as the loop filter for intra frames and proposes a scheme to solve the above problems. It aims to design a single CNN model with low redundancy to adapt to decoded frames with different qualities and ensure consistency. To adapt to reconstructions with different qualities, both reconstruction and QP are taken as inputs. After training, the obtained model is compressed to reduce redundancy. To ensure consistency, dynamic fixed points (DFP) are adopted in testing CNN. Parameters in the compressed model are first quantized to DFP and then used for inference of CNN. Outputs of each layer in CNN are computed by DFP operations. Experimental results on JEM 7.0 report 3.14%, 5.21%, 6.28% BD-rate savings for luma and two chroma components with all intra configuration when replacing all traditional filters.
---
paper_title: Multi-modal/multi-scale convolutional neural network based in-loop filter design for next generation video codec

paper_content:

In this paper, we propose a novel in-loop filter design for video compression. Our approach aims to replace existing deblocking filter and SAO (Sample Adaptive Offset) of HEVC standard with multi-modal/multi-scale convolutional neural network (MMS-net). The proposed CNN architecture consists of two sub-networks of different scales. An input image is down-sampled first and restored through the lower scale network, then the output image from it is fed into higher scale network concatenated with the original input image. Moreover, to boost the restoration performance, the proposed architecture utilizes information resides in the coded sequence. Specifically, the compression parameters from coding tree units (CTU) are exploited as input to CNN, which helps to alleviate blocking artifacts on the reconstructed images. In the experiments, our method reduces the average BD-rate by 4.55% and 8.5%, respectively, compared with the conventional neural network based approach [1] and HEVC reference software HM16.7 [2] in ‘All Intra — Main’ configuration.
---
paper_title: Sample Adaptive Offset in the HEVC Standard

paper_content:

This paper provides a technical overview of a newly added in-loop filtering technique, sample adaptive offset (SAO), in High Efficiency Video Coding (HEVC). The key idea of SAO is to reduce sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category. The offset of each category is properly calculated at the encoder and explicitly signaled to the decoder for reducing sample distortion effectively, while the classification of each sample is performed at both the encoder and the decoder for saving side information significantly. To achieve low latency of only one coding tree unit (CTU), a CTU-based syntax design is specified to adapt SAO parameters for each CTU. A CTU-based optimization algorithm can be used to derive SAO parameters of each CTU, and the SAO parameters of the CTU are inter leaved into the slice data. It is reported that SAO achieves on average 3.5% BD-rate reduction and up to 23.5% BD-rate reduction with less than 1% encoding time increase and about 2.5% decoding time increase under common test conditions of HEVC reference software version 8.0.
---
paper_title: Residual Highway Convolutional Neural Networks for in-loop Filtering in HEVC

paper_content:

High efficiency video coding (HEVC) standard achieves half bit-rate reduction while keeping the same quality compared with AVC. However, it still cannot satisfy the demand of higher quality in real applications, especially at low bit rates. To further improve the quality of reconstructed frame while reducing the bitrates, a residual highway convolutional neural network (RHCNN) is proposed in this paper for in-loop filtering in HEVC. The RHCNN is composed of several residual highway units and convolutional layers. In the highway units, there are some paths that could allow unimpeded information across several layers. Moreover, there also exists one identity skip connection (shortcut) from the beginning to the end, which is followed by one small convolutional layer. Without conflicting with deblocking filter (DF) and sample adaptive offset (SAO) filter in HEVC, RHCNN is employed as a high-dimension filter following DF and SAO to enhance the quality of reconstructed frames. To facilitate the real application, we apply the proposed method to I frame, P frame, and B frame, respectively. For obtaining better performance, the entire quantization parameter (QP) range is divided into several QP bands, where a dedicated RHCNN is trained for each QP band. Furthermore, we adopt a progressive training scheme for the RHCNN where the QP band with lower value is used for early training and their weights are used as initial weights for QP band of higher values in a progressive manner. Experimental results demonstrate that the proposed method is able to not only raise the PSNR of reconstructed frame but also prominently reduce the bit-rate compared with HEVC reference software.
---
paper_title: Adaptive Loop Filtering for Video Coding

paper_content:

Adaptive loop filtering for video coding is to minimize the mean square error between original samples and decoded samples by using Wiener-based adaptive filter. The proposed ALF is located at the last processing stage for each picture and can be regarded as a tool to catch and fix artifacts from previous stages. The suitable filter coefficients are determined by the encoder and explicitly signaled to the decoder. In order to achieve better coding efficiency, especially for high resolution videos, local adaptation is used for luma signals by applying different filters to different regions or blocks in a picture. In addition to filter adaptation, filter on/off control at coding tree unit (CTU) level is also helpful for improving coding efficiency. Syntax-wise, filter coefficients are sent in a picture level header called adaptation parameter set, and filter on/off flags of CTUs are interleaved at CTU level in the slice data. This syntax design not only supports picture level optimization but also achieves a low encoding latency. Simulation results show that the ALF can achieve on average 7% bit rate reduction for 25 HD sequences. The run time increases are 1% and 10% for encoders and decoders, respectively, without special attention to optimization in C++ code.
---
paper_title: Learning for Video Compression

paper_content:

One key challenge to learning-based video compression is that motion predictive coding, a very effective tool for video compression, can hardly be trained into a neural network. In this paper, we propose the concept of PixelMotionCNN (PMCNN) which includes motion extension and hybrid prediction networks. PMCNN can model spatiotemporal coherence to effectively perform predictive coding inside the learning network. On the basis of PMCNN, we further explore a learning-based framework for video compression with additional components of iterative analysis/synthesis and binarization. The experimental results demonstrate the effectiveness of the proposed scheme. Although entropy coding and complex configurations are not employed in this paper, we still demonstrate superior performance compared with MPEG-2 and achieve comparable results with H.264 codec. The proposed learning-based scheme provides a possible new direction to further improve compression efficiency and functionalities of future video coding.
---
paper_title: Unsupervised Learning of Video Representations using LSTMs

paper_content:

We use Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.
---
paper_title: Full Resolution Image Compression with Recurrent Neural Networks

paper_content:

This paper presents a set of full-resolution lossy image compression methods based on neural networks. Each of the architectures we describe can provide variable compression rates during deployment without requiring retraining of the network: each network need only be trained once. All of our architectures consist of a recurrent neural network (RNN)-based encoder and decoder, a binarizer, and a neural network for entropy coding. We compare RNN types (LSTM, associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study "one-shot" versus additive reconstruction architectures and introduce a new scaled-additive framework. We compare to previous work, showing improvements of 4.3%-8.8% AUC (area under the rate-distortion curve), depending on the perceptual metric used. As far as we know, this is the first neural network architecture that is able to outperform JPEG at image compression across most bitrates on the rate-distortion curve on the Kodak dataset images, with and without the aid of entropy coding.
---
paper_title: Video (language) modeling: a baseline for generative models of natural videos

paper_content:

We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequence, the model discovers both spatial and temporal correlations which are useful to represent complex deformations and motion patterns. The models we propose are largely borrowed from the language modeling literature, and adapted to the vision domain by quantizing the space of image patches into a large dictionary. We demonstrate the approach on both a filling and a generation task. For the first time, we show that, after training on natural videos, such a model can predict non-trivial motions over short video sequences.
---
paper_title: DeepCoder: A deep neural network based video compression

paper_content:

Inspired by recent advances in deep learning, we present the DeepCoder — a Convolutional Neural Network (CNN) based video compression framework. We apply separate CNN nets for predictive and residual signals respectively. Scalar quantization and Huffman coding are employed to encode the quantized feature maps (fMaps) into binary stream. We use the fixed 32 × 32 block in this work to demonstrate our ideas, and performance comparison is conducted with the well-known H.264/AVC video coding standard with comparable rate-distortion performance. Here distortion is measured using Structural Similarity (SSIM) because it is more close to perceptual response.
---
paper_title: CNN oriented fast PU mode decision for HEVC hardwired intra encoder

paper_content:

The number of intra prediction modes in High Efficiency Video Coding (HEVC) has been increased up to 35. To the end of alleviating the complexity of intra coding, we bring in the convolution neural network (CNN) to obtain the candidate modes of current PU and adopt the corner detection algorithm to further reduce the candidate modes. The virtues of proposed algorithm include: Firstly, our algorithm skip the rough PU mode decision (RMD) process and get the candidate list from CNN directly. In other words, the computations are relaxed in our algorithm. Secondly, the inputs of CNN in proposed algorithm merely contain the source image pixels and quantization parameter (QP), this feature makes it friendly to high parallel hardwired encoder. As compared with HM-15.0, experiments show that our algorithm decreases the intra coding time by 27.92% while the corresponding BDBR augment is 1.15%. At last but not least, our algorithm possesses a stable coding performance. In specific, for the most sensitive sequence (Class F), our algorithm could save 27.10% intra coding time with 2.01% BDBR increase.
---
paper_title: Reducing Complexity of HEVC: A Deep Learning Approach

paper_content:

High Efficiency Video Coding (HEVC) significantly reduces bit-rates over the preceding H.264 standard but at the expense of extremely high encoding complexity. In HEVC, the quad-tree partition of coding unit (CU) consumes a large proportion of the HEVC encoding complexity, due to the brute-force search for rate-distortion optimization (RDO). Therefore, this paper proposes a deep learning approach to predict the CU partition for reducing the HEVC complexity at both intra-and inter-modes, which is based on convolutional neural network (CNN) and long-and short-term memory (LSTM) network. First, we establish a large-scale database including substantial CU partition data for HEVC intra-and inter-modes. This enables deep learning on the CU partition. Second, we represent the CU partition of an entire coding tree unit (CTU) in the form of a hierarchical CU partition map (HCPM). Then, we propose an early-terminated hierarchical CNN (ETH-CNN) for learning to predict the HCPM. Consequently, the encoding complexity of intra-mode HEVC can be drastically reduced by replacing the brute-force search with ETH-CNN to decide the CU partition. Third, an early-terminated hierarchical LSTM (ETH-LSTM) is proposed to learn the temporal correlation of the CU partition. Then, we combine ETH-LSTM and ETH-CNN to predict the CU partition for reducing the HEVC complexity at inter-mode. Finally, experimental results show that our approach outperforms other state-of-the-art approaches in reducing the HEVC complexity at both intra-and inter-modes.
---
paper_title: CU Partition Mode Decision for HEVC Hardwired Intra Encoder Using Convolution Neural Network

paper_content:

The intensive computation of High Efficiency Video Coding (HEVC) engenders challenges for the hardwired encoder in terms of the hardware overhead and the power dissipation. On the other hand, the constrains in hardwired encoder design seriously degrade the efficiency of software oriented fast coding unit (CU) partition mode decision algorithms. A fast algorithm is attributed as VLSI friendly, when it possesses the following properties. First, the maximum complexity of encoding a coding tree unit (CTU) could be reduced. Second, the parallelism of the hardwired encoder should not be deteriorated. Third, the process engine of the fast algorithm must be of low hardware- and power-overhead. In this paper, we devise the convolution neural network based fast algorithm to decrease no less than two CU partition modes in each CTU for full rate-distortion optimization (RDO) processing, thereby reducing the encoder’s hardware complexity. As our algorithm does not depend on the correlations among CU depths or spatially nearby CUs, it is friendly to the parallel processing and does not deteriorate the rhythm of RDO pipelining. Experiments illustrated that, an averaged 61.1% intraencoding time was saved, whereas the Bjontegaard-Delta bit-rate augment is 2.67%. Capitalizing on the optimal arithmetic representation, we developed the high-speed [714 MHz in the worst conditions (125 °C, 0.9 V)] and low-cost (42.5k gate) accelerator for our fast algorithm by using TSMC 65-nm CMOS technology. One accelerator could support HD1080p at 55 frames/s real-time encoding. The corresponding power dissipation was 16.2 mW at 714 MHz. Finally, our accelerator is provided with good scalability. Four accelerators fulfill the throughput requirements of UltraHD-4K at 55 frames/s.
---
paper_title: A Joint Compression Scheme of Video Feature Descriptors and Visual Content.

paper_content:

High-efficiency compression of visual feature descriptors has recently emerged as an active topic due to the rapidly increasing demand in mobile visual retrieval over bandwidth-limited networks. However, transmitting only those feature descriptors may largely restrict its application scale due to the lack of necessary visual content. To facilitate the wide spread of feature descriptors, a hybrid framework of jointly compressing the feature descriptors and visual content is highly desirable. In this paper, such a content-plus-feature coding scheme is investigated, aiming to shape the next generation of video compression system toward visual retrieval, where the high-efficiency coding of both feature descriptors and visual content can be achieved by exploiting the interactions between each other. On the one hand, visual feature descriptors can achieve compact and efficient representation by taking advantages of the structure and motion information in the compressed video stream. To optimize the retrieval performance, a novel rate-accuracy optimization technique is proposed to accurately estimate the retrieval performance degradation in feature coding. On the other hand, the already compressed feature data can be utilized to further improve the video coding efficiency by applying feature matching-based affine motion compensation. Extensive simulations have shown that the proposed joint compression framework can offer significant bitrate reduction in representing both feature descriptors and video frames, while simultaneously maintaining the state-of-the-art visual retrieval performance.
---
paper_title: Joint Rate-Distortion Optimization for Simultaneous Texture and Deep Feature Compression of Facial Images

paper_content:

The explosion of surveillance cameras in smart cites and the increasing demand of low latency visual analysis have pushed the horizon from the traditional image/video compression to feature compression. Due to the recent advances of face recognition, we investigate the simultaneous compression of facial images and deep features, which is demonstrated to be beneficial in terms of the whole system performance including visual quality and recognition accuracy. Herein, we propose the Texture-Feature-Quality-Index (TFQI) to measure the ultimate utility of the facial images based on automatic visual analysis and monitoring. Furthermore, based on TFQI, a bit allocation scheme is proposed to optimally allocate the given bits for images and features, such that the overall coding performance can be optimized. The proposed scheme is validated using the standard face verification benchmark, Labeled Faces in the Wild (LFW). Better rate-TFQI and rate-Accuracy performance compared to the traditional texture coding can be achieved, especially in the scenario of low bit-rate coding.
---
paper_title: Complexity-Constrained H.264 Video Encoding

paper_content:

In this paper, a joint complexity-distortion optimization approach is proposed for real-time H.264 video encoding under the power-constrained environment. The power consumption is first translated to the encoding computation costs measured by the number of scaled computation units consumed by basic operations. The solved problem is then specified to be the allocation and utilization of the computational resources. A computation allocation model (CAM) with virtual computation buffers is proposed to optimally allocate the computational resources to each video frame. In particular, the proposed CAM and the traditional hypothetical reference decoder model have the same temporal phase in operations. Further, to fully utilize the allocated computational resources, complexity-configurable motion estimation (CAME) and complexity-configurable mode decision (CAMD) algorithms are proposed for H.264 video encoding. In particular, the CAME is performed to select the path of motion search at the frame level, and the CAMD is performed to select the order of mode search at the macroblock level. Based on the hierarchical adjusting approach, the adaptive allocation of computational resources and the fine scalability of complexity control can be achieved.
---

Title: Image and Video Compression with Neural Networks: A Review

Section 1: INTRODUCTION OF NEURAL NETWORK AND IMAGE/VIDEO COMPRESSION
Description 1: Introduction to the basic concepts and development history of neural networks, as well as the frameworks and basic technique development for block-based image coding and hybrid video coding framework.

Section 2: Neural Network
Description 2: Discussion on the invention, structure, and functionality of neural networks, focusing on interdisciplinary research between neuroscience and mathematics.

Section 3: Image and Video Compression
Description 3: Core techniques in image and video compression, including transform and prediction, and comparison of various coding frameworks such as JPEG, MPEG-2, H.264/AVC, and HEVC.

Section 4: PROGRESS OF NEURAL NETWORK BASED IMAGE COMPRESSION
Description 4: Review on the development of neural network-based image compression techniques, including detailed descriptions of methods based on Multilayer Perceptron (MLP), Random Neural Networks, Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Generative Adversarial Networks (GAN).

Section 5: Multi-layer Perceptron based Image Coding
Description 5: Exploration of the historical development and application of MLP in image compression, focusing on representative methods and structures.

Section 6: Random Neural Network based Image Coding
Description 6: Description of Random Neural Networks and their specific applications and adaptations for image compression.

Section 7: Convolutional Neural Network based Coding
Description 7: Detailed discussion on the application of CNNs in lossy image compression, and the techniques employed to overcome the challenges related to quantization and rate-distortion optimization.

Section 8: Recurrent Neural Network based Coding
Description 8: Review of RNN architectures with memory for image compression, highlighting progressive and spatially adaptive compression methods.

Section 9: Generative Adversarial Network based Coding
Description 9: Examination of GAN-based methods for image compression, focusing on how adversarial loss improves the subjective quality of decoded images.

Section 10: ADVANCEMENT OF VIDEO CODING WITH NEURAL NETWORKS
Description 10: An overview of improvements in video coding modules, such as intra prediction, inter-prediction, quantization, entropy coding, and loop filtering, through the integration of various deep learning models.

Section 11: Intra Prediction Techniques using Neural Networks
Description 11: Focused discussion on network-based enhancements for intra prediction in HEVC, including specific techniques and performance results.

Section 12: Neural Network based Inter Prediction
Description 12: Exploration of deep learning-based improvements for inter prediction, including motion estimation refinement and fractional-pixel reference generation.

Section 13: Neural Network based Quantization and Entropy Coding for Video Coding
Description 13: Discussion on neural network enhancements for quantization and entropy coding processes in video compression.

Section 14: Neural Network based Loop Filtering
Description 14: Review of CNN-based loop filtering techniques aimed at artifact reduction, including specific model architectures and their coding gains.

Section 15: New Video Coding Frameworks Based on Neural Network
Description 15: Examination of novel video coding frameworks leveraging neural network models, diverging from traditional hybrid frameworks.

Section 16: OPTIMIZATION TECHNIQUES FOR IMAGE AND VIDEO COMPRESSION
Description 16: Overview of optimization techniques for image and video compression leveraging neural networks, including fast mode-decision algorithms.

Section 17: CONCLUSIONS AND OUTLOOK
Description 17: Summarizes the reviewed neural network-based image and video compression techniques, identifies advantages and challenges, and discusses future research directions.