---
abstract: 'This paper proposes a novel approach to regularize the *ill-posed* blind image deconvolution (blind image deblurring) problem using deep generative networks. We employ two separate deep generative models — one trained to produce sharp images while the other trained to generate blur kernels from lower-dimensional parameters. To deblur, we propose an alternating gradient descent scheme operating in the latent lower-dimensional space of each of the pretrained generative models. Our experiments show excellent deblurring results even under large blurs and heavy noise. To improve the performance on rich image datasets not *well learned* by the generative networks, we present a modification of the proposed scheme that governs the deblurring process under both generative and classical priors.'
bibliography:
- 'egbib.bib'
title: Blind Image Deconvolution using Pretrained Generative Priors
---

Introduction and Related Work {#sec:introduction}
=============================

Blind image deblurring aims to recover a true image $i$ and a blur kernel $k$ from blurry and possibly noisy observation $y$. For a uniform and spatially invariant blur, it can be mathematically formulated as $$\label{eq:bid}
y = i \otimes k + n,$$ where $\otimes $ is a convolution operator and $n$ is an additive Gaussian noise. In its full generality, the inverse problem is severely ill-posed as many different instances of $i$, and $k$ fit the observation $y$ [@campisi2016blind; @kundur1996blind]. To resolve between multiple instances, priors are introduced on images and/or blur kernels in the image deblurring algorithms. Priors assume an *a priori* model on the true image/blur kernel or both. These natural structures expect images or blur kernels to be sparse in some transform domain; see, for example, [@chan1998total; @fergus2006removing; @levin2009understanding; @hu2010single; @zhang2011sparse; @cai2009blind]. Some of the other penalty functions to improve the conditioning of the blind image deblurring problem are low-rank [@ren2016image], and total variation based priors [@pan2014motion]. A recently introduced dark channel prior [@pan2016blind] also shows promising results; it assumes a sparse structure on the dark channel of the image, and exploits this structure in an optimization program [@xu2011image] to solve the blind image deblurring problem. Other works include extreme channel priors [@yan2017image], outlier robust deblurring [@dong2017blind], learned data fitting [@pan2017learning], and discriminative prior based blind image deblurring approaches [@li2018learning]. Although generic and applicable to multiple applications, these engineered models are not very effective as many unrealistic images also fit the prior model [@hand2017global].

\
\

Recently deep learning based blind image deblurring approaches have shown impressive results due to their power of learning from large training data [@hradivs2015convolutional; @nah2016deep; @schuler2016learning; @xu2017learning; @nimisha2017blur; @kupyn2017deblurgan]. Generally, these deep learning based approaches invert the forward acquisition model of blind image deblurring via end-to-end training of deep neural networks in a supervised manner. The main drawback of this end-to-end deep learning approach is that it does not explicitly take into account the knowledge of forward map , but rather learns implicitly from training data. Consequently, the deblurring is more sensitive to changes in the blur kernels, images, or noise distributions in the test set that are not representative of the training data, and often requires expensive retraining of the network for a competitive performance [@lucas2018using].

Meanwhile, neural network based implicit generative models such as generative adversarial networks (GANs) [@goodfellow2014generative] and variational autoencoders (VAEs) [@kingma2013auto] have found much success in modeling complex data distributions especially that of images. Recently, GANs and VAEs have been used for blind image deblurring but only in an end-to end manner [@xu2017learning; @nimisha2017blur; @kupyn2017deblurgan] , which is completely different from our approach as will be discussed in detail. These methods show competitive performance, but since these generative model based approaches are end-to-end they suffer from the same draw backs as other deep learning based debluring approaches. On the other hand, pretrained generative models have recently been employed as regularizers to solve inverse problems in imaging including compressed sensing [@bora2017compressed; @shah2018solving], image inpainting [@yeh2017semantic], Fourier ptychography [@shamshad2018deep], and phase retrieval [@shamshad2018robust; @hand2018phase]. However the applicability of these pretrained generative models in blind image deblurring is relatively unexplored. Recently [@gandelsman2018double] employ a combination of multiple untrained deep generative models and show their effectiveness on various image layer decomposition tasks including image water mark removal, image dehazing, image segmentation, and transparency separation in images and videos. Different from their approach, we show the effectiveness of our blind image deblurring method by leveraging trained generative models for images and blurs.

In this work, we use the expressive power of pretrained GANs and VAEs to tackle the challenging problem of blind image deblurring. Our experiments in Figure \[fig:intro-results\] confirm that integrating deep generative priors in the image deblurring problem enables a far more effective regularization yielding sharper and visually appealing deblurred images. Specifically, our main contributions are

-   To the best of our knowledge, this is the first instance of utilizing pretrained generative models for tackling challenging problem of blind image deblurring.

-   We show that simple gradient descent approach assisted with generative priors is able to recover true image and blur kernel, to with in the range of respective generative models, from blurry image.

-   We investigate a modification of the loss function to allow the recovered image some leverage/slack to deviate from the range of the image generator. This modification effectively addresses the performance limitation due to the range of the generator.

-   Our experiments demonstrate that our approach produce superior results when compared with traditional image priors and unlike deep learning based approaches does not require expensive retraining for different noise levels.

Problem Formulation and Proposed Solution {#sec:Problem-Formulation}
=========================================

We assume the image $i \in \R^n$ and blur kernel $k \in \R^n$ in are members of some structured classes $\mathcal{I}$ of images, and $\mathcal{K}$ of blurs, respectively. For example, $\setI$ may be a set of celebrity faces and $\setK$ comprises of motion blurs. A representative sample set from both classes $\setI$ and $\setK$ is employed to train a generative model for each class. We denote $G_{\mathcal{I}}: \mathbb{R}^l \rightarrow \mathbb{R}^n$ and $G_{\mathcal{K}}: \mathbb{R}^m \rightarrow \mathbb{R}^n$ as the generators for class $\setI$, and $\setK$, respectively. Given low-dimensional inputs $z_i \in \mathbb{R}^l$, and $z_k \in \mathbb{R}^m$, the pretrained generators $G_{\setI}$ and $G_{\setK}$ generate new samples $G_{\setI}(z_i)$, and $G_{\setK}(z_k)$ that are representative of the classes $\setI$ and $\setK$, respectively. Once trained, the weights of the generators are fixed. To recover the sharp image and blur kernel $(i,k)$ from the blurred image $y$ in , we propose minimizing the following objective function $$\begin{aligned}
\label{eq:Optimization-Ambient}
(\hat{i},\hat{k}) :=  \underset{\substack{i \in\text{Range}(G_{\setI}) \\k \in\text{Range}(G_{\setK})}}{\text{argmin}} \ \|y - i \otimes k \|^2, \end{aligned}$$ where $\|\cdot\|$ is the $\ell_2$-distance, Range($G_\setI$) and Range($G_\setK$) is the set of all the images and blurs that can be generated by $G_\setI$ and $G_\setK$, respectively. In words, we want to find an image $i$ and a blur kernel $k$ in the range of their respective generators, that best explain the forward model . Ideally, the range of a pretrained generator comprises of only the samples drawn from the probability distribution of the training image or blur class. Constraining the solution $(\hat{i},\hat{k})$ to lie only in generator ranges forces the solution to be the members of classes $\setI$ and $\setK$.

The minimization program in can be equivalently formulated in the lower dimensional, latent representation space as follows: $$\begin{aligned}
\label{eq:Optimization-latent}
(\hat{z}_i, \hat{z}_k) = \underset{z_i \in \R^l, z_k \in \R^m}{\text{argmin}}
\ 
\| y - G_{\mathcal{I}}(z_i) \otimes G_\setK(z_k) \|^2.\end{aligned}$$ This optimization program can be thought of as tweaking the latent representation vectors $z_i$ and $z_k$, (input to the generators $G_{\setI}$, and $G_{\setK}$, respectively) until these generators generate an image $i$ and blur kernel $k$ whose convolution comes as close to $y$ as possible. Incorporating the fact that latent representation vectors $z_i$, and $z_k$ are assumed to be coming from standard Gaussian distributions, we further augment the measurement loss in with $\ell_2$ penalty terms on the latent representations. The resultant optimization program is then $$\label{eq:regularized-program}
\underset{z_i \in \R^l, z_k \in \R^m}{\text{argmin}}\ \| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2+ \gamma\| z_i \|^2 + \lambda\| z_k \|^2,$$ where $\gamma$ and $\lambda$ are free scalar parameters. For brevity, we denote the objective function above by $\setL(z_i,z_k)$. Importantly, the weights of the generators are always fixed as they enter into this algorithm as pretrained models. To minimize this non-convex objective, we begin by initializing $z_i$ and $z_k$ by sampling from standard Gaussian distribution, and resort to an alternating gradient descent algorithm by taking a gradient step in one of these while fixing the other to find a minima $(\hat{z}_i, \hat{z}_k)$. To avoid being stuck in a not good enough local minima, we restart the algorithm with a new random initialization (Random Restarts) when the measurement loss in does not reduce sufficiently after reasonably many iterations. We dubbed proposed deblurring algorithm as *Deep Deblur* and denote blurry image deblurred via *Deep Deblur* as $\hat{\textit{{i}}}_\text{DD}$.The estimated deblurred image and the blur kernel are acquired by a forward pass of the solutions $\hat{z}_i$ and $\hat{z}_k$ through the generators $G_{\mathcal{I}}$ and $G_{\mathcal{K}}$. Mathematically, $(\hat{i},\hat{k}) = ( G_\mathcal{I}(\hat{z}_i), G_\mathcal{K}(\hat{z}_k))$.

Problem Formulation {#sec:Problem-Formulation}
===================

We assume the image $i \in \R^n$ and blur kernel $k \in \R^n$ in are members of some structured classes $\mathcal{I}$ of images, and $\mathcal{K}$ of blurs, respectively. For example, $\setI$ may be a set of celebrity faces and $\setK$ comprises of motion blurs. A representative sample set from both classes $\setI$ and $\setK$ is employed to train a generative model for each class. We denote by the mappings $G_{\mathcal{I}}: \mathbb{R}^l \rightarrow \mathbb{R}^n$ and $G_{\mathcal{K}}: \mathbb{R}^m \rightarrow \mathbb{R}^n$, the generators for class $\setI$, and $\setK$, respectively. Given low-dimensional inputs $z_i \in \mathbb{R}^l$, and $z_k \in \mathbb{R}^m$, the pretrained generators $G_{\setI}$, and $G_{\setK}$ generate new samples $G_{\setI}(z_i)$, and $G_{\setK}(z_k)$ that are representative of the classes $\setI$, and $\setK$, respectively. Once trained, the weights of the generators are fixed. To recover the sharp image, and blur kernel $(i,k)$ from the blurred image $y$ in , we propose minimizing the following objective function $$\begin{aligned}
\label{eq:Optimization-Ambient}
(\hat{i},\hat{k}) :=  \underset{\substack{i \in\text{Range}(G_{\setI}) \\k \in\text{Range}(G_{\setK})}}{\text{argmin}} \ \|y - i \otimes k \|^2, \end{aligned}$$ where Range($G_\setI$) and Range($G_\setK$) is the set of all the images and blurs that can be generated by $G_\setI$ and $G_\setK$, respectively. In words, we want to find an image $i$ and a blur kernel $k$ in the range of their respective generators, that best explain the forward model . Ideally, the range of a pretrained generator comprises of only the samples drawn from the distribution of the image or blur class. Constraining the solution $(\hat{i},\hat{k})$ to lie only in generator ranges, therefore, implicitly reduces the solution ambiguities inherent to the ill-posed blind deconvolution problem, and forces the solution to be the members of classes $\setI$, and $\setK$.

The minimization program in can be equivalently formulated in the lower dimensional, latent representation space as follows $$\begin{aligned}
\label{eq:Optimization-latent}
(\hat{z}_i, \hat{z}_k) = \underset{z_i \in \R^l, z_k \in \R^m}{\text{argmin}}
\ 
\| y - G_{\mathcal{I}}(z_i) \otimes G_\setK(z_k) \|^2.\end{aligned}$$ This optimization program can be thought of as tweaking the latent representation vectors $z_i$, and $z_k$, (input to the generators $G_{\setI}$, and $G_{\setK}$, respectively) until these generators generate an image $i$ and blur kernel $k$ whose convolution comes as close to $y$ as possible.

The optimization program in is obviously non-convex owing to the bilinear convolution operator, and non-linear deep generative models. We resort to an alternating gradient descent algorithm to find a local minima $(\hat{z}_i, \hat{z}_k)$. Importantly, the weights of the generators are always fixed as they enter into this algorithm as pretrained models. At a given iteration, we fix $z_i$ and take a descent step in $z_k$, and vice verse. The gradient step in each variable involves a forward and backward pass through the generator networks. Section \[proposedapproach\] talks about the back propagation, and gives explicit gradient forms for descent in each $z_i$ and $z_k$ for this particular algorithm.

The estimated deblurred image and the blur kernel are acquired by a forward pass of the solutions $\hat{z}_i$ and $\hat{z}_k$ through the generators $G_{\mathcal{I}}$ and $G_{\mathcal{K}}$. Mathematically, $(\hat{i},\hat{k}) = ( G_\mathcal{I}(\hat{z}_i), G_\mathcal{K}(\hat{z}_k))$.

Image Deblurring Algorithm {#sec:proposedapp}
==========================

Our approach requires pretrained generative models $G_\setI$ and $G_\setK$ for classes $\setI$ and $\setK$, respectively. We use both GANs and VAEs as generative models on the clean images and blur kernels.

![image](./figures/block_diag.png){width="75.00000%"}

Naive Deblurring {#backprop}
----------------

To deblur an image $y$, a simplest possible strategy is to find an image closest to $y$ in the range of the given generator $G_{\setI}$ of clean images. Mathematically, this amounts to solving the following optimization program $$\begin{aligned}
 \label{eq:naive-deblur}
\underset{z_i \in \R^l}{\text{argmin}} \  \| y - G_\setI(z_i) \|, \quad \quad \hat{i} = G_\setI(\hat{z}_i), \end{aligned}$$ where we emphasize again that in the optimization program above, the weights of the generator $G_\setI$ are fixed (pretrained). Although non-convex, a local minima $\hat{z}_i$ can be achieved via gradient descent implemented using the back propagation algorithm. The recovered image $\hat{i}$ is obtained by a forward pass of $\hat{z}_i$ through the generative model $G_{\setI}$. Expectedly, this approach fails to produce reasonable recovery results; see Figure \[fig:Back-Propagation\]. The reason being that this back projection approach completely ignores the knowledge of the forward blur model in . We now address this shortcoming by including the forward model and blur kernel in the objective (\[eq:naive-deblur\]).

Deconvolution using Deep Generative Priors {#proposedapproach}
------------------------------------------

We discovered in the previous section that simply finding a clean image close to the blurred one in the range of the image generator $G_{\setI}$ is not good enough. A more natural and effective strategy is to instead find a pair consisting of a clean image and a blur kernel in the range of $G_{\setI}$, and $G_{\setK}$, respectively, whose convolution comes as close to the blurred image $y$ as possible. As outlined in Section \[sec:Problem-Formulation\], this amounts to minimizing the measurement loss $$\label{eq:measurement-loss}
\| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2,$$ over both $z_i$, and $z_k$, where $\otimes$ is the convolution operator. Incorporating the fact that latent representation vectors $z_i$, and $z_k$ are assumed to be coming from standard Gaussian distributions in both the adversarial learning and variational inference framework, outlined in Section \[generative\], we further augment the measurement loss in with $\ell_2$ penalty terms on the latent representations. The resultant optimization program is then $$\label{eq:regularized-program}
\underset{z_i \in \R^l, z_k \in \R^m}{\text{argmin}}\ \| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2+ \gamma\| z_i \|^2 + \lambda\| z_k \|^2,$$ where $\lambda$, and $\gamma$ are free scalar parameters. For brevity, we denote the objective function above by $\setL(z_i,z_k)$. To minimize this non-convex objective, we begin by initializing $z_i$, and $z_k$ as standard Gaussian vectors, and then take a gradient step in one of these while fixing the other. To avoid being stuck in a not good enough local minima, we may restart the algorithm with a new random initialization (Random Restarts) when the measurement loss in does not reduce sufficiently after reasonably many iterations. Algorithm \[alg:generative-prior-deblurring\] formally introduces the proposed alternating gradient descent scheme. Henceforth, we will denote the image deblurred using Algorithm \[alg:generative-prior-deblurring\] by $\hat{i}_\text{DD}$.

Beyond the Range of Generator {#sec:proposed-method2}
-----------------------------

As described earlier, the optimization program implicitly constrains the deblurred image to lie in the range of the generator $G_{\setI}$. This may leads to some artifacts in the deblurred images when the generator range does not completely span the set $\setI$. This inability of the generator to completely learn the image distribution is often evident in case of more rich and complex natural images. In such cases, it makes more sense to not strictly constrain the recovered image to come from the range of the generator, and rather also explore images a bit outside the range. To accomplish this, we propose minimizing the measurement loss of images inside the range exactly as in together with the measurement loss $\| y - i \otimes G_\setK(z_k) \|^2$ of images not necessarily within the range. The in-range image $G_{\setI}(z_i)$, and the out-range image $i$ are then tied together by minimizing an additional penalty term, $\text{Range Error(i)} := \| i - G_\setI(z_i) \|^2$. The idea is to strictly minimize the range error when pretrained generator has effectively learned the image distribution, and afford some slack when it is not the case. The amount of slack can be controlled by tweaking the weights attached with each loss term in the final objective. Finally, to guide the search of a best deblurred image beyond the range of the generator, one of the conventional image priors such as total variation measure $\|\cdot\|_{\text{tv}}$ is also introduced. This leads to the following optimization program

$$\begin{aligned}
 \label{eq:opt-admm}
\underset{i, z_i, z_k}{\text{argmin}}  \ &
\| y - i \otimes G_\setK(z_k) \|^2 + 
\tau \| i - G_\setI(z_i) \|^2 + \zeta\| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2 + \rho\|i\|_{\text{tv}}.\end{aligned}$$

All of the variables are randomly initialized, and the objective is minimized using gradient step in each of the unknowns, while fixing the others. The computations of gradients is very similar to the steps outlined in Section \[proposedapproach\]. We take the solution $\hat{i}$, and $G(z_k)$ as the deblurred image, and the recovered blur kernel. The iterative scheme is formally given in Algorithm \[alg:generative+classical-prior-deblurring\]. For future references, we will denote the recovered image using Algorithm \[alg:generative+classical-prior-deblurring\] by $\hat{i}_\text{DDS}$.

![[]{data-label="fig:proposed_approach"}](./figures/block_diag.png){width="90.00000%"}

A more natural and effective strategy is to instead find a pair consisting of a clean image and a blur kernel in the range of $G_{\setI}$, and $G_{\setK}$, respectively, whose convolution comes as close to the blurred image $y$ as possible; see Figure \[fig:proposed\_approach\]. As outlined in Section \[sec:Problem-Formulation\], this amounts to minimizing the measurement loss $$\label{eq:measurement-loss}
\| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2,$$ over both $z_i$ and $z_k$. Incorporating the fact that latent representation vectors $z_i$, and $z_k$ are assumed to be coming from standard Gaussian distributions in both the adversarial learning and variational inference framework, we further augment the measurement loss in with $\ell_2$ penalty terms on the latent representations. The resultant optimization program is then $$\label{eq:regularized-program}
\underset{z_i \in \R^l, z_k \in \R^m}{\text{argmin}}\ \| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2+ \gamma\| z_i \|^2 + \lambda\| z_k \|^2,$$ where $\gamma$ and $\lambda$ are free scalar parameters. For brevity, we denote the objective function above by $\setL(z_i,z_k)$. Importantly, the weights of the generators are always fixed as they enter into this algorithm as pretrained models. To minimize this non-convex objective, we begin by initializing $z_i$ and $z_k$ as standard Gaussian vectors, and then take a gradient step in one of these while fixing the other. To avoid being stuck in a not good enough local minima, we may restart the algorithm with a new random initialization (Random Restarts) when the measurement loss in does not reduce sufficiently after reasonably many iterations. Algorithm \[alg:generative-prior-deblurring\] formally introduces the proposed alternating gradient descent scheme. Henceforth, we will denote the image deblurred using Algorithm \[alg:generative-prior-deblurring\] by $\hat{i}_1$.

\[alg:AltGradDescent\]

**Input:** $y$, $G_\setI$ and $G_\setK$\
**Output:** Estimates $\hat{i}_1$ and $\hat{k}$\
**Initialize:**\
$z_i^{(0)} \sim  \setN(0,I), ~~ z_k^{(0)} \sim  \setN(0,I)$ (\[eq:regularized-program\])\
(\[eq:regularized-program\])\
$\hat{i}_1  \leftarrow G_\setI(z^{(T)}_i), \hat{k}  \leftarrow G_\setK(z^{(T)}_k)$

\[alg:generative-prior-deblurring\]

Beyond the Range of Generator {#sec:proposed-method2}
-----------------------------

\[alg:admm\]

**Input:** $y$, $G_\setI$ and $G_\setK$\
**Output:** Estimates $\hat{i}_2$ and $\hat{k}$\
**Initialize:**\
$z_i^{(0)} \sim  \setN(0,I), z_k^{(0)} \sim  \setN(0,I), i^{(0)} \sim  \setN(0.5,10^{-2}I)$ (\[eq:opt-admm\])\
(\[eq:opt-admm\])\
(\[eq:opt-admm\])\
$\hat{i}_2  \leftarrow {i^{(T)}}, \hat{k}  \leftarrow G_\setK(z_k^{(T)})$

\[alg:generative+classical-prior-deblurring\] \[algorithm:admm\]

Beyond the Range of Generator {#sec:proposed-method2}
-----------------------------

As described earlier, the optimization program implicitly constrains the deblurred image to lie in the range of the generator $G_{\setI}$. This may lead to some artifacts in the deblurred images when the generator range does not completely span the set $\setI$. In such case, it makes more sense to not strictly constrain the recovered image to come from the range of the generator, and rather also explore images a bit outside the range. To accomplish this, we propose minimizing the measurement loss of images inside the range exactly as in together with the measurement loss $\| y - i \otimes G_\setK(z_k) \|^2$ of images not necessarily within the range. The in-range image $G_{\setI}(z_i)$ and the out-range image $i$ are then tied together by minimizing an additional penalty term, $\text{Range Error(i)} := \| i - G_\setI(z_i) \|^2$. The idea is to strictly minimize the range error when pretrained generator has effectively learned the image distribution, and afford some slack otherwise. Finally, to guide the search of a best deblurred image beyond the range of the generator, one of the conventional image priors such as total variation measure $\|\cdot\|_{\text{tv}}$ is also introduced. This leads to the following optimization program $$\begin{aligned}
 \label{eq:opt-admm}
\underset{i, z_i, z_k}{\text{argmin}}  \ &
\| y - i \otimes G_\setK(z_k) \|^2 + 
\tau \| i - G_\setI(z_i) \|^2 
 +\zeta\| y - G_\setI(z_i) \otimes G_\setK(z_k) \|^2 + \rho\|i\|_{\text{tv}}\end{aligned}$$ All of the variables are randomly initialized, and the objective is minimized using gradient step in each of the unknowns, while fixing the others. We take the solution $\hat{i}$, and $G(\hat{z}_k)$ as the deblurred image, and the recovered blur kernel. We dubbed this approach as *Deep Deblur with Slack* (DDS) and the image deblurred using this approach is referred to as $\hat{\textit{i}}_\text{DDS}$.

Experimental Results {#experiments}
====================

In this section, we provide a comprehensive set of experiments to evaluate the performance of *Deep Deblur* and *Deep Deblur with Slack* against iterative and deep learning based baseline methods. We also evaluate performance under increasing noise and large blurs. In all experiments, we use noisy blurred images, generated by convolving images $i$, and blurs $k$ from their respective test sets and adding 1$\%$ [^1] Gaussian noise (unless stated otherwise). The choice of free parameters in both algorithms for each dataset are provided in the supplementary material.

Implementation Details
----------------------

**Datasets**: We choose three image datasets. First dataset, SVHN, consists of house number images from Google street view. A total of 531K images, each of dimension $32\times32 \times 3$, are available out of which 30K are held out as test set. Second dataset, Shoes [@yu2014fine] consists of 50K RGB examples of shoes, resized to $64 \times 64 \times 3$. We leave $1000$ images for testing and use the rest as training set. Third dataset, CelebA, consists of relatively more complex images of celebrity faces. A total of 200K, each center cropped to dimension $64 \times 64 \times 3$, are available out of which 22K are held out as a test set. A motion blur dataset is generated consisting of small to very large blurs of lengths varying between 5 and 28; following strategy given in [@boracchi2012modeling]. We generate 80K blurs out of which 20K is held out as a test set.

**Generative Models**: We choose VAE as the generative model for SVHN images and motion blurs. For Shoes and CelebA, the generative model $G_{\setI}$ is the default deep convolutional generative adversarial network (DCGAN) [@salimans2016improved]. Further details on architectures of generative models are provided in the supplementary material.

**Baseline Methods**: Among the conventional algorithms using engineered priors, we choose dark prior (DP) [@pan2016blind], extreme channel prior (EP) [@yan2017image], outlier handling (OH) [@dong2017blind], and learned data fitting (DF) [@pan2017learning] based blind deblurring as baseline algorithms. We optimized the parameters of these methods in each experiment to obtain the best possible baseline results. Among driven approaches for deblurring, we choose [@hradivs2015convolutional] that trains a convolutional neural network (CNN) in an end-to-end manner, and [@kupyn2017deblurgan] that trains a neural network (DeblurGAN) in an adversarial manner. Deblurred images from these baseline methods will be referred to as $i_\text{DP}$, $i_\text{EP}$, ${i_\text{OH}}$, ${i_\text{DF}}$, $i_\text{CNN}$ and $i_\text{DeGAN}$.

Deblurring Results under Pretrained Generative Priors {#sec:Exps-PretrainedPriors}
-----------------------------------------------------

The central limiting factor in the *Deep Deblur* performance is the ability of the generator to represent the (original, clean) image to be recovered. As pointed out earlier that often the generators are not fully expressive (cannot generate new representative samples) on a rich/complex image class such as face images compared to a compact/simple image class such as numbers. Such a generator mostly cannot *adequately* represent a new image in its range. Since *Deep Deblur* strictly constrains the recovered image to lie in the range of image generator, its performance depends on how well the range of the generator spans the image class. Given an arbitrary test image $i_{\text{test}}$ in the set $\setI$, the closest image $i_{\text{range}}$, in the range of the generator, to $i_{\text{test}}$ is computed by solving the following optimization program $$\begin{aligned}
z_{\text{test}} := \underset{z} {\text{argmin}}  \|i_{\text{test}}-G_{\setI}(z)\|^2,  \quad i_{\text{range}} = G_{\setI}(z_{\text{test}})\end{aligned}$$ We solve the optimization program by running $10,000(6,000)$ gradient descent steps with a step size of $0.001(0.01)$ for CelebA(SVHN). Parameters for Shoes are the same as CelebA.

A more expressive generator leads to a better deblurring performance as it can well represent an arbitrary original (clean) image $i_{\text{test}}$ leading to a smaller mismatch $ \text{range error} := \|i_{\text{test}} - i_{\text{range}}\|$ to the corresponding range image $i_{\text{range}}$.

### Impact of Generator Range on Image Deblurring

To judge the proposed deblurring algorithms independently of generator range limitations, we present their deblurring performance on range image $i_{\text{range}}$; we do this by generating a blurred image $y = i_{\text{range}} \otimes k + n$ from an image $i_{\text{range}}$ already in the range of the generator; this implicitly removes the range error as now $i_{\text{test}} = i_{\text{range}}$. We call this range image deblurring, where the deblurred image is obtained using *Deep Deblur*, and is denoted by $\hat{i}_{\text{range}}$. For completeness, we also assess the overall performance of the algorithm by deblurring arbitrary blurred images $y = i_{\text{test}} \otimes k +n$, where $i_{\text{test}}$ is not necessarily in the range of the generator. Unlike above, the overall error in this case accounts for the range error as well. We call this arbitrary image deblurring, and specifically the deblurred image is obtained using *Deep Deblur*, and is denoted by $\hat{i}_\text{DD}$. Figure \[fig:range-images\] shows a qualitative comparison between $i_{\text{test}}$, $i_{\text{range}}$, and $\hat{i}_\text{DD}$ on CelebA dataset. It is clear that the recovered image $\hat{i}_\text{DD}$ is a good approximation of the range image, $i_{\text{range}}$, indicating the limitation of the image generative network.

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_19\_x\_hat\_from\_test.png]{} (64,0)

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_19\_x\_hat\_from\_test.png]{} (64,0)

[./figures/CelebA/“Original Images”/x\_orig\_19.png]{} (64,0)

\

*Deep Deblur with Slack* mitigates the range error by not strictly constraining the recovered image to lie in the range of the image generator, for details, see Section \[sec:proposed-method2\]. As shown in Figure \[fig:celebA-results\], estimate $\hat{i}_\text{DDS}$ of true image $i_\text{test}$ from blurry observations is close to $i_\text{test}$ instead of $i_\text{range}$, thus mitigating the range issue.

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_92\_x\_hat\_from\_test.png]{} (55,0)

[./figures/SVHN/“Original Images”/x\_orig\_92.png]{} (55,0)

\

\

\
\

### Deblurring Results on CelebA, Shoes and SVHN

**Qualitative results on CelebA**: Figure \[fig:celebA-results\] gives a qualitative comparison between $i$, $i_{\text{range}}$, $\hat{i}_\text{DD}$, $\hat{i}_\text{DDS}$, and baseline approaches on CelebA and Shoes dataset. The deblurred images under engineered priors are qualitatively a lot inferior than the deblurred images $\hat{i}_\text{DD}$, and $\hat{i}_\text{DDS}$ under the proposed generative priors, especially under large blurs. On the other hand, the end-to-end training based approaches CNN, and DeblurGAN perform relatively better, however, the well reputed CNN is still displaying over smoothed images with missing edge details, etc compared to our results $\hat{i}_\text{DDS}$. DeblurGAN, though competitive, is outperformed by the proposed *Deep Deblur with Slack* by more than 1.5dB as shown in Table \[table:psnr-ssim-results\]. The images $\hat{i}_\text{DD}$ are sharp and with well defined facial boundaries and markers owing to the fact they strictly come from the range of the generator, however, in doing so these images might end up changing some image features such as expressions, nose, etc. On a close inspection, it becomes clear that how well $\hat{i}_\text{DD}$ approximates $i_{\text{test}}$ roughly depends (see, images specifically in Figure \[fig:range-images\]) on how close $i_{\text{range}}$ is to $i_{\text{test}}$ exactly. While as $\hat{i}_\text{DDS}$ are allowed some leverage, and are not strictly confined to the range of the generator, they tend to agree more closely with the ground truth. We go on further by utilizing pretrained PGGAN [@karras2017progressive] in *Deep Deblur* by convolving sampled images with large blurs ($30 \times 30$); see Figure \[fig:pg-gan-results\]. It has been observed that pre-trained generators struggle at higher resolutions [@athar2018latent], so we restrict our results at $128 \times 128$ resolution. In Figure \[fig:pg-gan-results\], it can be seen that under expressive generative priors our approach exceeds all other baseline methods recovering fine details from extremely blurry images.

**Qualitative Results on SVHN**: Figure \[fig:svhn-results\] gives qualitative comparison between proposed and baseline methods on SVHN dataset. Here the deblurring under classical priors again clearly under performs compared to the proposed image deblurring results $\hat{i}_\text{DD}$. CNN also continues to be inferior, and the DeblurGAN also shows artifacts. We do not include the results $\hat{i}_\text{DDS}$ in these comparison as $\hat{i}_\text{DD}$ already comprehensively outperform the other techniques on this dataset. The convincing results $\hat{i}_\text{DD}$ are a manifestation of the fact that unlike the relatively complex CelebA and Shoes datasets, the simpler image dataset SVHN is effectively spanned by the range of the image generator.

\

**Quantitative Results**: Quantitative results for CelebA, Shoes[^2] and SVHN using peak-signal-to-noise ratio (PSNR) and structural-similarity index (SSIM) [@wang2004image], averaged over 80 respective test set images, are given in Table \[table:psnr-ssim-results\]. On CelebA and Shoes, the results clearly show a better performance of *Deep Deblur with Slack*, on average, compared to all baseline methods. On SVHN, the results show that *Deep Deblur* outperforms all competitors. The fact that *Deep Deblur* performs more convincingly on SVHN is explained by observing that the range images $i_{\text{range}}$ in SVHN are quantitatively much better compared to range images of CelebA and Shoes.

### Robustness against Noise and Large Blurs

**Robustness against Noise**: Figure \[fig:psnr-ssim-blursize\] gives a quantitative comparison of the deblurring obtained via *Deep Deblur* (the free parameters $\lambda$, $\gamma$ and random restarts in the algorithm are fixed as before), and baseline methods CNN, DeblurGAN (trained on fixed 1% noise level and on varying 1-10% noise levels) in the presence of Gaussian noise. We also include the performance of deblurred range images $\hat{i}_{\text{range}}$, introduced in Section \[sec:Exps-PretrainedPriors\], as a benchmark. Conventional prior based approaches are not included as their performance substantially suffers on noise compared to other approaches. On the vertical axis, we plot the PSNR and on the horizontal axis, we vary the noise strength from 1 to 10%. In general, the quality of deblurred range images (expressible by the generators) $\hat{i}_{\text{range}}$ under generative priors surpasses other algorithms on both CelebA, and SVHN. This in a way manifests that under expressive generative priors, the performance of our approach is far superior. The quality of deblurred images $\hat{i}_\text{DD}$ under generative priors with arbitrary (not necessarily in the range of the generator) input images is the second best on SVHN, however, it under performs on the CelebA dataset; the most convincing explanation of this performance deficit is the range error (not as expressive generator) on the relatively complex/rich images of CelebA. The end-to-end approaches trained on fixed 1% noise level display a rapid deterioration on other noise levels. Comparatively, the ones trained on 1-10% noise level, expectedly, show a more graceful performance. Qualitative results under heavy noise are depicted in Figure \[fig:results-heavynoise\]. Our deblurred image $\hat{i}_\text{DD}$ visually agrees better with $i_{\text{test}}$ than other methods.

**Robustness against Large Blurs**: Figure \[fig:results-largeblur\] shows the deblurred images obtained from a very blurry face image. The deblurred image $\hat{i}_\text{DDS}$ using *Deep Deblur with Slack* is able to recover the true face from a completely unrecognizable face. The classical baseline algorithms totally succumb to such large blurs. The quantitative comparison against end-to-end neural network based methods CNN, and DeblurGAN is given in Figure \[fig:psnr-ssim-blursize\]. We plot the blur size against the average PSNR for both Shoes, and CelebA datasets. On both datasets, deblurred images $\hat{i}_\text{DDS}$ convincingly outperforms all other techniques. For comparison, we also add the performance of $\hat{i}_{\text{range}}$. Excellent deblurring under large blurs can also be seen in Figure \[fig:pg-gan-results\] for PGGAN. To summarize, the end-to-end approaches begin to lag a lot behind our proposed algorithms when the blur size increases. This is owing to the firm control induced by the powerful generative priors on the deblurring process in our newly proposed algorithms.

Conclusion {#sec:conc}
==========

This paper proposes a novel framework for blind image deblurring that uses deep generative networks as priors rather than in a conventional end-to-end manner. We report convincing deblurring results under the generative priors in comparison to the existing methods. A thorough discussion on the possible limitations of this approach on more complex images is presented along with an effective remedy to address these shortcomings. Our main contribution lies in introducing pretrained generative model in solving blind deconvolution. Introducing more expressive generative models with new novel architectures would improve the performance. We leave these exciting directions for future work.

Supplementary Stuff
===================

Algorithm Parameters
--------------------

The choice of free parameters for SVHN, Shoes and CelebA of *Deep Deblur* and *Deep Deblur with Slack* are given in Table \[table:alg1-param\] and \[table:alg2-param\].

Both algorithms were implemented in Tensorflow and the code will be made publicly available.

Generative Models
=================

The generative model of SVHN images is a trained VAE with the network architecture described in Table \[table:vae-architectures\]. The dimension of the latent space of VAE is fixed at 100, and training is carried out on SVHN with a batch size of 1500, and a learning rate of $10^{-5}$ using the Adam optimizer. After training, the decoder part is extracted as the desired generative model $G_{\setI}$. For CelebA and Shoes dataset, the generative model $G_{\setI}$ is the default deep convolutional generative adversarial network (DCGAN)[@salimans2016improved].

The generative model of motion blur dataset is a trained VAE with the network architecture given in Table \[table:vae-architectures\]. This VAE is trained using Adam optimizer with latent dimension 50, batch size 5, and learning rate $10^{-5}$. After training, the decoder part is extracted as the desired generative model $G_{\setK}$.

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_10\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_10\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_10.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_18\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_18\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_18.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_26\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_26\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_26.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_48\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_48\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_48.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_73\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_73\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_73.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_53\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_53\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_53.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_35\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_35\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_35.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_02\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_02\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_02.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_79\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_79\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_79.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_06\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_06\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_06.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_40\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_40\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_40.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_63\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_63\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_63.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_72\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_72\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_72.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_39\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_39\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_39.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_00\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_00\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_00.png]{}

\

[./figures/CelebA/“deblurring - 1perc noise - 10RR”/deblurring\_07\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“deblurring - admm - 1perc noise - 10RR”/deblurring\_07\_x\_hat\_from\_test.png]{}

[./figures/CelebA/“Original Images”/x\_orig\_07.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_03\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_03.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_04\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_04.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_07\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_07.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_10\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_10.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_17\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_17.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_31\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_31.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_37\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_37.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_44\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_44.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_57\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_57.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_61\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_61.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_68\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_68.png]{}

\

[./figures/SVHN/“deblurring - 1perc noise - 10RR”/deblurring\_76\_x\_hat\_from\_test.png]{}

[./figures/SVHN/“Original Images”/x\_orig\_76.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_12\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_12\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_12.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_24\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_24\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_24.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_26\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_26\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_26.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_31\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_31\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_31.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_33\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_33\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_33.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_40\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_40\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_40.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_35\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_35\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_35.png]{}

\

[./figures/Shoes/“deblurring - 1perc noise - 10RR”/deblurring\_37\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“deblurring - admm - 1perc noise - 10RR”/deblurring\_37\_x\_hat\_from\_test.png]{}

[./figures/Shoes/“Original Images”/x\_orig\_37.png]{}

\

\
\
\

[^1]: For an image scaled between 0 and 1, Gaussian noise of $1\%$ translates to Gaussian noise with standard deviation $\sigma = 0.01$ and mean $\mu=0$.

[^2]: For qualitative results, see supplementary material.
