In recent years Convolutional Neural Networks (CNN) have been used extensively for Superresolution (SR). In this paper, we use inverse problem and sparse representation solutions to form a mathematical basis for CNN operations. We show how a single neuron is able to provide the optimum solution for inverse problem, given a low resolution image dictionary as an operator. Introducing a new concept called Representation Dictionary Duality, we show that CNN elements (filters) are trained to be representation vectors and then, during reconstruction, used as dictionaries. In the light of theoretical work, we propose a new algorithm which uses two networks with different structures that are separately trained with low and high coherency image patches and show that it performs faster compared to the state-of-the-art algorithms while not sacrificing from performance. Recent years have witnessed an increased demand for superresolution (SR) algorithms. Increased number of video devices boosted the need for displaying high quality videos online with lower bandwidth. In addition, the social media required the storage of videos and images with lowest possible size for server optimization. Other areas include 4K video displaying from Full HD broadcasts, increasing the output size for systems that have limited sized sensors, such as medical imaging, thermal cameras and surveillance systems.SR algorithms aim to generate high-resolution (HR) image from single or ensemble of lowresolution (LR) images. The observation model of a real imaging system relating a high resolution image to the low resolution observation frame can be given as DISPLAYFORM0 where H models the blurring effects, S models the downsampling operation, and n models the system noise. The solution to this problem seeks a minimal energy of an energy functional comprised of the fidelity of the estimated imagef to the observational image f .State-of-the art algorithms that are addressing SR problem can be collected under Dictionary learning based methods (DLB) and Deep learning based methods (DLM) categories. Although SR problem is an inverse problem by nature, performance of other methods such as Bayesian and Example based methods have been surpassed which is the reason why they are not included in this work. Also the SR problem has never been directly dealt with inverse problem solutions as in BID3 BID4 DLB are generally solving optimization problems with sparsity constraints such as BID20 BID21 and L 2 norm regularization as in BID19 . The main concern of DLB is creation of a compact dictionary for reconstruction of high resolution (HR) image. Although useful, DLB methods become heavy and slow algorithms as reconstruction performance increases. Recent advances on GPUs have fueled the usage of convolutional neural networks (CNNs) for SR problem. CNN based algorithms such as BID5 and BID9 have used multi-layered networks which have successfully surpassed DLB methods in terms of run speed and performance. State-of-the art algorithms also use Perceptual Loss (PL) to generate new textures from LR images BID11 . By uniting PL and generative networks, photo realistic images can be generated BID10 . PL minimization based algorithms are visually superior to MSE minimization based ones. Stability of such algorithms have been improved since they have been first proposed BID7 . Although, stability issue is not yet completely addressed for generative networksIn BID1 authors have described representation learning as a manifold learning for which a higher dimensional data is represented compactly in a lower dimensional manifold. They have discussed that the variations in the input space is captured by the representations, for which we are explaining the mechanism at work.Though CNNs are successful for SR problem experimentally, their mathematical validation is still lacking. We summarized the contributions of this work.• We show that neurons solve an Iterative Shrinkage Thresholding (IST) equation during training for which the operator is dictionary matrix constructed from LR training data. The solution yields a representation vector as the neuron filters. Contrary to the discussion in literature for which an encoder-decoder structure is needed to obtain and use representations, we claim that the filters themselves become the representations.• We describe a new concept namely Representation Dictionary Duality (RDD) and show that neuron filters act as representation vectors during training phase. Then in the testing phase, filters start acting as dictionaries upon which the HR reconstruction is made layer by layer. This is a concept which helps us analyze CNNs with sparse representation and inverse problem mathematics.• After analyzing a neuron with inverse problem and DLB solutions and discussing how the entire network operates during training, we propose a new network structure which is able to recover certain details better, faster without sacrificing overall performance.Rest of the paper organized as follows: in section 2 we refer to related literature for different areas of research. Section 3 ties previous work into our analysis of CNNs . In section 4 we propose a new network for SR problem. In section 5 we give experimentation results.2 RELATED WORK 2.1 ANALYTIC APPROACHES Solution to eq. 1 is inherently ill-conditioned since a multiplicity of solutions exist for any given LR pixel. Thus proper prior regularization for the high resolution image is crucial. The regularization of the inversion is provided with a function , reg, which promotes the priori information from the desired output, reg takes different forms ranging from L 0 norm, Tikhonov regularization to orthogonal decomposition of the estimate. Denoting the SH matrix in eq. 2 by K, the regularized solution is given byf DISPLAYFORM1 In BID4 DISPLAYFORM2 Where a class of proximity operators are defined, the special function for the case of L 1 regularization is soft thresholding function also known as shrinkage operator. DISPLAYFORM3 Notice that K T (g − Kf n−1 ) is the negative gradient of data fidelity term in the original formulation. Therefore the solution for the inverse problem using IST iterations is obtained in a gradient descent type method thresholded by Moreau proximity mapping which is also named as Proximal Landweber Iterations. BID4 have proposed the usage of non-quadratic regularization constraints that promote sparsity by the help of an orthonormal (or overcomplete) basis ϕ l of a Hilbert space. For the problem defined in eq. 2 it is proposed to use a functional φ b, p as DISPLAYFORM4 For the case when p = 1, a straightforward variational equation can be obtained in an iterative way. DISPLAYFORM5 Iterations over the set of basis functions can be carried out in one formula DISPLAYFORM6 where DISPLAYFORM7 which can be seen as a method to file the elements of x in the direction of ϕ l . Daubechies et. al. have proven that the solution obtained by iterating f is the global minimum of the solution space. The solution will reach to an optimum point if K is a bounded operator satisfying ||Kf || ≤ C||f || for any vector f and some constant C.We will use this result in proving that neurons in a CNN architecture are able to reach to the optimum solution for SR problem by solving for the exact same eq. 7. A similar work is conducted by BID8 . They have proposed a Learned IST algorithm which can be seen as a time unfolded recurrent neural network. Later BID2 have discussed that LISTA and their own algorithms that extend LISTA are not mere approximations for an iterative algorithm but themselves are full featured sparse coders.Our work diverges from theirs in showing how a convolutional neural network is able to learn image representation and reconstruction for SR problem inside network parameters. We will unite inverse problem approaches, DLM and DLB methods in a representation-dictionary duality concept. We have proven that a neuron is able to solve an inverse problem optimally. By introducing RDD we have shown that CNN layers act as sparse representation solvers. We have proposed a method that addresses the texture recovery better. Experiments have shown that RDD is valid and proposed network recovers some texture components better and faster than state of the art algorithms while not sacrificing performance and speed. In the future we plan to investigate a content-aware aggregation method which might perform better than simple averaging. We will investigate ways of jointly training or optimizing two networks and including aggregation step inside a unified network. In parallel we are investigating a better network structure for texture recovery. Also we are going to incorporate the initial upsampling step into the network by allowing the network to learn its own interpolation kernels.A VISUAL RESULTS
After proving that a neuron acts as an inverse problem solver for superresolution and a network of neurons is guarantied to provide a solution, we proposed a double network architecture that performs faster than state-of-the-art. Discusses using neural networks for super-resolution A new architecture for solving image super-resolution tasks, and an analysis aiming to establish a connection between CNNs for solving super resolution and solving sparse regularized inverse problems.