Deep Learning (DL) algorithms based on Generative Adversarial Network (GAN) have demonstrated great potentials in computer vision tasks such as image restoration. Despite the rapid development of image restoration algorithms using DL and GANs, image restoration for specific scenarios, such as medical image enhancement and super-resolved identity recognition, are still facing challenges. How to ensure visually realistic restoration while avoiding hallucination or mode- collapse? How to make sure the visually plausible results do not contain hallucinated features jeopardizing downstream tasks such as pathology identification and subject identification?
 Here we propose to resolve these challenges by coupling the GAN based image restoration framework with another task-specific network. With medical imaging restoration as an example, the proposed model conducts additional pathology recognition/classification task to ensure the preservation of detailed structures that are important to this task. Validated on multiple medical datasets, we demonstrate the proposed method leads to improved deep learning based image restoration while preserving the detailed structure and diagnostic features. Additionally, the trained task network show potentials to achieve super-human level performance in identifying pathology and diagnosis.
 Further validation on super-resolved identity recognition tasks also show that the proposed method can be generalized for diverse image restoration tasks. Image restoration is an essential computer vision task and a widely applied technique. Recently there are increasing interests and significant progresses in this area enabling more realistic image super-resolution BID6 ; BID17 ; ; BID2 ; BID43 BID39 , in-painting BID35 ; BID38 ; BID37 ; BID31 and denoising BID35 ; BID41 a) . With the development of image restoration technologies, various applications can be applied in different verticals to reach the unfulfilled needs.Among all the image restoration applications, restoration in medical imaging is one of the most challenging tasks. Image restoration in medical imaging is important and attractive, since it enables imaging in more desirable conditions, e.g. imaging with faster protocols BID29 , cheaper devices and lower radiation BID26 , etc. However, medical image restoration requires a tougher evaluation than restoring natural images. It does not only require sharper and visually realistic restoration, but also requires accurate image completion without altering any pathological features or affecting any diagnostic qualities/properties. Therefore medical image restoration can be a benchmark task for related image restoration techniques.Within this decade, image restoration technique has been rapidly growing by incorporating various prior information into solving the ill-posed inverse imaging task. The prior information evolves from using sparse representation assumption BID23 , enforcing low-rank analysis BID7 to more recently using deep learning based priors BID33 or models BID42 . However there are still several challenges and limitations for existing algorithms: 1) Pixel-wise losses for deep learning do not consider non-local structural information which leads to blurred and not visually plausible restoration BID17 .2) Generative Adversarial Network (GAN) BID11 based methods significantly improve the results to generate visually realistic restoration BID17 . However GANs ensure the consistency to a learned distribution but do not necessarily guarantee the visually plausible solution exactly matches the corresponding ground truth.3) It is still possible that hallucination or mode-collapses may happen while minimizing loss function designed in improved GAN frameworks BID10 , BID0 .4) The discriminator network regularizes on general image distribution and visual quality , but it does not consider what are the key characteristic features such as pathologies, contrasts and image identification that the model needs to preserve for restoring an image.These challenges are critical for vertical applications such as medical imaging and surveillance where not only the visual property but also the fidelity of the recovered details matters for key purposes of pathology or recognizing faces.To solve the problems and challenges for realistic and accurate image restoration, we propose the task-GAN which extends GAN based image restoration framework and includes 3 networks: a Generator, a Discriminator and a Task-specific Network. The new task-specific network predicts the pathology recognition or face identity from both the ground truth images and the restored images. It helps to regularize the training of generator and complement the adversarial loss of GAN to ensure the output images better approximate the ground truth images. Task-GAN both achieves realistic visual quality and preserves the important task-specific features/properties, which are related to the end goal for medical imaging restoration and super-resolution face restoration.The contribution of this work are:• We propose a Task Generative Adversarial Network framework (Task-GAN) to ensure both visually plausible and more accurate (medical/face) image restoration.• A Task Network and a task-driven loss are introduced to ensure the preservation of visual details important to the downstream tasks, and more importantly it regularizes the image restoration to be more accurate both quantitatively and qualitatively.• The method is validated on two in-vivo clinical medical imaging datasets across different modalities, including Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). Additionally , the generalization of the proposed method is further evaluated on a super-resolution face restoration dataset.• Both quantitative and qualitative evaluations were conducted, including rigorous evaluation by human experts (radiologists) to ensure the image restoration quality and preservation of important visual features.• Results demonstrate the superiority of the proposed method in image restoration and also show the potential of applying the trained task network for super-human level automatic classification/diagnosis.• Theory behind the method is further discussed. More justification on how the proposed method improves GAN to approximate one-to-one mapping.The way of how the proposed Task-GAN improves the image restoration may lead to better model design for other applications. Results on in-vivo medical imaging datasets demonstrate the superior performance of the proposed algorithm on improved image restoration. The proposed task-GAN achieves this by coupling adversarial training with the training of the task-specific network. Detailed contribution of the task-GAN is explained in the figure 4.In comparison, the task of the image restoration is to learn a non-linear mapping from low-quality images in the measurement domain to its corresponding high-quality images in a different highquality domain containing visually realistic images. Shown in figure 4(a ), in addition, the recognition of image is a space separation of features/labels along different dimensions that can be orthogonal to the quality dimensions.In comparison, as is shown in FIG4 ( b), conventional learning strategy learns the image restoration task by regression, which may fail to generate realistic restoration. The learning is usually based on the minimization of an averaged distance penalty which ensures robustness but lead to unrealistic restoration such as blurring. Additionally, the averaged solution is also likely to be away from the distribution of visually plausible solutions that falls out of the high-quality image space as is shown in the figure.GAN-based approach on one hand overcomes this by further enforcing an adversarial loss with a Discriminator network which ensures to generate realistic restoration following the distribution of the target high-quality images. As the figure 4(c ) shows, the solution is no longer an simple average but pushed into the space of visually realistic high-quality images. However , on the other hand, the discriminator only regularizes the output samples to follow the distribution but ignores the inter-sample relationship. For example , it cannot avoid hallucinations or mode-collapse, where the restored images may be over-similar or undesirably add/remove important visual features. As is shown in the figure, the restored image can have a different label as the ground-truth which fails the purpose of image restoration. We can picture the hallucinations or mode-collapse as a "shrinking" of solution space.To avoid the possible mode-collapse and ensure a 1-to-1 mapping, various improved GAN models and cost functions have been proposed. For example, Cycle- GAN Zhu et al. (2017) incorporate a cyclic relationship to improve the mapping. However, cyclic relationship does not necessarily lead to exact mappings. The inter-sample relationship as well as the important feature labels can be swapped while still satisfying the cyclic relationship. The illustrating image can be found in the appendix. For example, in the figure, one task label is altered while the cyclic loss is not affected. This may lead to mode-collapse, or specifically a failed image restoration leading to misclassified pathology/normality for medical imaging applications. The consequences of the restoration errors can be huge for medical imaging applications since they can directly lead to mis-diagnosis or overdiagnosis. We can picture the mislabeling or mode-collapse as a "twisting" of solution space. This "twisting" maintains well within visually-plausible space, however severely changes the positioning around the decision boundary of task-label space. More details of the reasoning and visualization will be place in the appendix.Differently, task-GAN here regularizes both the inter-sample relationship and the sample-label relationship. As is shown in figure 4(d), accurate mapping can be generated with the mixed loss regularization:1 ) pixel-level supervision so the restored image is closer to the ground truth,2) Adversarial loss regularization so that the restored image is within the high-quality space consisting of visually realistic images 3) the task-specific loss that ensure the restored image still preserve the important feature of interests, aka the same labels. In other words, the combination regularization enforce the solution to fall onto the intersection of the manifold preserving pixel-level similarity, distribution consistency and important visual labels. In the view of inter-sample relationship, the task regularization stop the inter-sample relationship to any visual plausible but destructive "shrinking" or "twisting" around the boundary of task-label space, which ensures more accurate mappings. In this paper, we proposed an improved design of GAN, Task-GAN, which includes a new taskspecific network and corresponding task-specific loss for training GAN based image restoration. Task-GAN is demonstrated to boost the performance of image restoration while preserving important features. Medical imaging applications are used as primary examples, which is one of the most challenging restoration applications since it requires not only realistic restoration, but also high-fidelity as well as accurate classification for subtle diagnostic features. Super-resolution face restoration is used to show the proposed method generalize to natural image applications such as super-resolving face images, where face identity need to be preserved.The proposed method is demonstrated to achieve superior performance compared with GAN on both image quality metrics and task-specific feature preservation (e.g. pathological features, face identity features, etc.). Based on visual inspection from human experts (clinicians/radiologists), anatomical and diagnostic features are preserved better and fewer artifacts are introduced. The trained task network also shows potentials for super-human level diagnosis tasks.Task-GAN further extends the regularization of adversarial training. The mixed loss balances between content similarity, distribution consistency and preserving important features for the given tasks. It results in more accurate image restoration with better visual similarity and avoids modecollapse and hallucinations. Intuitively, task-GAN enforces the solution fall into proper manifold, prevents any alternation ("shrinking" and "twisting") of the restoration from the correct solution space, and preserves both inter-sample relationship and feature-of-interest.In the future, we will explore further improvements in the design of networks and task formulation. The proposed technique is also valuable to other challenging restoration applications that require realistic restoration and preserving distinguishable details for down-stream tasks.
Couple the GAN based image restoration framework with another task-specific network to generate realistic image while preserving task-specific features. A novel method of Task-GAN of image coupling that couples GAN and a task-specific network, which alleviates to avoid hallucination or mode collapse. The authors propose to augment GAN-based image restoration with another task-specific branch, such as classification tasks, for further improvement.