---
abstract: 'Low light image enhancement is an important challenge for the development of robust computer vision algorithms. The machine learning approaches to this have been either unsupervised, supervised based on paired dataset or supervised based on unpaired dataset. This paper presents a novel deep learning pipeline that can learn from both paired and unpaired datasets. Convolution Neural Networks (CNNs) that are optimized to minimize standard loss, and Generative Adversarial Networks (GANs) that are optimized to minimize the adversarial loss are used to achieve different steps of the low light image enhancement process. Cycle consistency loss and a patched discriminator are utilized to further improve the performance. The paper also analyses the functionality and the performance of different components, hidden layers, and the entire pipeline.'
author:
- 
bibliography:
- 'references.bib'
title: A Retinex based GAN Pipeline to Utilize Paired and Unpaired Datasets for Enhancing Low Light Images
---

low-light image enhancement, retinex theory, generative adversarial networks, cycle consistency

Introduction
============

Recently there has been a great and growing interest in the field of computer vision and image processing. Due to the abundance of visual data and the higher computational power of portable devices, computer vision applications have been integrated into our day-to-day lives [@computer-vision-survey]. However, many of the researches done on image transformation and interpretation are focused on well-lit images. Well-lit images can only be obtained under natural lighting during a fraction of the day or in the presence of artificial lighting during the rest of the day. There are situations where artificial lighting is not feasible (energy concerns, environmental concerns, obstruct natural lightning, etc.). Thus it is essential to identify low light images and enhance their features to guarantee the robustness of computer vision algorithms.

Low light images suffer from a range of issues such as low visibility, noise, colour distortion, etc. These issues prohibit them from being visually perceptible for the human eye and being useful for computer vision algorithms in terms of information richness. Enhancement of these low light images has been tried on hardware front (sensitive light-capturing mechanisms, longer exposure times, etc.) with varying degrees of success but at higher costs. The algorithmic approach to enhancing these images is considered to be an important research problem.

Algorithmic approaches to low light image enhancement are built upon the understanding of photonics, digital image properties, and biology related to human vision. Retinex theory of human vision [@Land1977] has been the defacto inspiration for most of the algorithmic pipelines in this domain. The algorithms can be broadly broken down into non-trainable algorithms which have a fixed activity based on programmer’s understanding of the process, trainable (learning) algorithms which learn the low light image enhancement from example images, and hybrids of both these approaches (which are most successful). When it comes to learning algorithms (or pipelines utilizing trainable sections) we see systems built with trainable filters on standard CNNs [@recent-advances-cnn] and GANs [@gan].

These learning algorithms learn from datasets of two forms – paired (containing a low light and well-lit captures of the same scene/object) and unpaired (containing sets of low light and high light images without a counterpart). The paired datasets are more informative but are difficult to obtain. They are relatively rare in number and the variety of scenes in them. In contrast, unpaired datasets are not as informative as the low light datasets, are easier to obtain, abundant, and contain a wide variety of scenes/objects. Existing solutions depend on only one of these two types.

This paper presents a DNN based algorithmic pipeline to enhance low light images with the following advantages.

-   The system utilizing both paired and unpaired datasets. Standard CNNs are used to learn from the paired datasets while GANs are used to learn from unpaired datasets.

-   The CNN and GAN architectures and their coordination are designed with the retinex model as an inspiration.

Related works
=============

### **Classical Algorithms** {#classical-algorithms .unnumbered}

Classical image processing algorithms are unsupervised algorithms that enhance low light images through well-formulated mathematical models. These algorithms are based on two schools of thought, namely intensity-based enhancement[@AHE] and gradient-based enhancement[@grad_enhance]. They are computationally efficient and simple. But they are not robust enough to be used over different conditions without manual calibration.

### **Retinex theory** {#retinex-theory .unnumbered}

The Retinex Theory [@Land1977] is a biologically-motivated theory based on the colour constancy property of the human visual system (HVS). It states that any object has a lighting independent property known as the reflectance and the perceived colour of the object occurs due to the illumination on these objects. Thus, the retinex based algorithms focus on the decomposition of the image to obtain the reflectance which represents the “true colour” of the object. Many variations of the original retinex model have been proposed.[@Land1977; @land1983recent; @wang2015variational] Recent works have made improvements to the retinex model for better performance and improved robustness [@ret_noise] while other techniques such as LIME [@lime] employs a similar idea for the decomposition and enhancement of illumination maps. The model proposed in this work is also motivated by the retinex model.

### **Deep Convolution Networks** {#deep-convolution-networks .unnumbered}

Deep learning techniques such as convolution neural networks (CNNs) [@lecun1999object] and autoencoder networks [@Ballard1987ModularLI] have been proved to perform better than classical algorithms in many image processing tasks [@imagenet; @cnn1]. In the field of low-light image enhancement, neural networks have been proposed in works such as LLNet [@llnet] (based on stacked auto-encoder), LLCNN [@llcnn] (based on the retinex theory), MSRnet [@msrnet] (based on log transforms) and  [@pipeline-nn] (based on wavelet transform). Recently, the Retinex-net [@deep-retinex-decomp] and [@see_in_dark] have been proposed. These are full frameworks for low light image enhancement, which brings impressive results. However, these techniques have certain shortcomings, the major one being that these architectures depend on training data in the form of paired datasets, which is difficult to obtain.

### **Adversarial Networks** {#adversarial-networks .unnumbered}

Generative Adversarial Network [@gan] have proven to perform sufficiently well for many supervised and unsupervised learning problems. In [@cycle-gan] the authors propose a model through which the need for paired images has been elevated and image translation between two domains can be done through cycle-consistence loss. These techniques have been applied to many other applications including dehazing, super-resolution, etc. Lately, it has been applied to low light image enhancement in EnlightenGAN [@en-gan] with promising results and this has motivated our GAN model.

### **Dataset** {#dataset .unnumbered}

The main requirement for many learning models is the existence of paired training data with low/normal-light images and several methods exist to collect such data. The images in LOL [@deep-retinex-decomp] dataset are captured in the daytime under normal light condition by changing the exposure time and ISO. The SID [@see_in_dark] dataset is generated by increasing the exposure time to generate well-lit images. However, it consists of raw sensor data under extremely low-light scenes, and this may limit its application for general low-light enhancement researches. Furthermore, due to the tedious experimental procedure, these datasets contain only a few images.

Supervised learning models depend on the availability of a large volume of training data but, the aforementioned techniques are neither efficient nor scalable. Thus synthetic image datasets are used as an alternative in many works [@brightening-train; @attention_guide]. However, synthetic low light images are created by processing images taken under normal light conditions, and these algorithms may not consider other factors. This results in performance variation on low-light images taken in real-world conditions and therefore may not be suitable for non-synthetic dark images.

The introduction of learning models such as cyclic-GANs has removed the dependency on paired datasets. Therefore, low light images present in [@Exdark; @deep-retinex-decomp] [^1] can be mixed with high exposure images to create a dataset of unpaired low/normal light image.

Proposed Method
===============

The proposed model splits the enhancement processes into multiple stages and each stage should fulfil one of the objectives specified below.

-   The image’s light level should be measured.

-   The image’s information should be extracted even in the poorly-light condition.

-   The image’s light should be increased while preserving and enhancing the information.

-   The noise and deformations introduced to the image during the enhancement process should be taken care of.

This work proposes a pipeline to achieve all those objectives through a series of neural networks. The first two objectives are achieved through the **decomp-net** (motivated by the retinex theory) which decomposes the images into the reflectance (colour information) and illumination (lighting information). The next two objectives are achieved through the **enhancement-net**. The first part of the process focuses on local neighbourhood of pixels while the second part focuses on both the local and global neighbourhoods.

The decomposition requires an understanding of how the images change under different lighting conditions. This relationship could be learned effectively using paired images (same scene captured under well-lit and low light condition). The enhancement requires an understanding of different types of scenes and objects in images. It is difficult to obtain a paired dataset for this task. Therefore we depend on an unpaired dataset. The overall architecture is given in [Fig. \[fig:full-system\]]{} and the following subsections will describe the individual components.

![image](figures/full_model.png){width="0.9\linewidth"}

Retinex decomposition network
-----------------------------

The first part of the pipeline performs retinex decomposition on the images. This retinex idea could be extended to digital images (where $S$ and $R$ are 2 spatial dimensional matrices with 3 colour channels and $I$ is a 2 spatial dimensional matrix with 1 channel) as $S = R \circ I$ where $\circ$ is the spatial element-wise multiplication. The images are decomposed into two parts as the reflectance $R$ and the illumination $I$.

-   Reflectance $R$ : This part has the colour information of the image. This is a 3 channel image of dimensions similar to the original image. We assume this part to be consistent over different lighting conditions for a single scene/object.

-   Illumination $I$: This is a single channel image that represents the lighting effect of an image.

We define the generators used in the translation $S_{low} \rightarrow S_{high}$ as $G_i$ and the generators used in the translation $S_{high} \rightarrow S_{low}$ as $F_i$. We define the NN decomposition of low light images as $G_1$ and the NN decomposition of high light images as $F_1$. But the symmetric training process ensures them both being the same $G_1 = F_1$. This is illustrated in [Fig. \[fig:full-system\]]{} and [Fig. \[fig:forward\_reverse\]]{}.

-   $G_1 : S_{low} \rightarrow [R_{low},I_{low}]$

-   $F_1 : S_{high} \rightarrow [R_{high},I_{high}]$

For this, we use Deep Retinex Decomposition Network proposed in [@deep-retinex-decomp]. Both $G_1$ and $F_1$ were trained using LOL dataset [@deep-retinex-decomp] which consists of well-lit and low light, coloured image pairs. 16 random patches of size $64\times64$ are sampled from an image. The NN was trained to minimize the custom loss defined in [@deep-retinex-decomp] with the adam[@adam] optimizer at a learning rate of 0.001 and a decay factor of 0.9 for 100 iterations.

Enhancement network
-------------------

The enhancement is done using a neural network. The architecture is inspired by U-Net [@unet] and its modifications over U-Net are given in [Table \[tab:unet\]]{}. This CNN operates on the concatenation of $R$ and $I$ as the input (unlike U-Net). This will output an enhanced version of the illumination map. Since it is difficult to find a dataset with low light and well-lit image pairs, using a paired dataset learning technique to train the enhancement network is not practical. Therefore, we use an unpaired dataset with a Generative Adversarial Network (GAN) to train the enhancement network.

            Parameter                        Value
  ----------------------------- -------------------------------
           Input size                  $256 \times 256$
     Down/upsampling factor            $\frac{1}{2} , 2$
   No. down/upsampling layers               $7 , 7$
   No channels (hidden layers)   4,128,256,512,512,512,512,512
        Conv kernel size                 $3 \times 3$

  : U-Net Specifications

\[tab:unet\]

![Block diagram of the generator components.[]{data-label="fig:forward_reverse"}](figures/forward.png){width=".9\linewidth"}

![Block diagram of the generator components.[]{data-label="fig:forward_reverse"}](figures/backward.png){width=".9\linewidth"}

We use the cycle consistency loss [@cycle-gan] to train the enhancement network with an unpaired dataset.

### Cycle consistency

To enhance the training process we use two GANs[@gan] that will generate well-lit images from low light images and vice versa. We use these GANs as illustrated in [Fig. \[fig:full-system\]]{} to preserve the cycle consistency using an adversarial loss [@cycle-gan]. This reduces the distance between the generated image and the expected image distributions.

The “cycle” consists of a forward pass $S_{low} \rightarrow S_{high'} \rightarrow S_{low''}$ and a backward pass $S_{low} \rightarrow S_{high'} \rightarrow S_{low''}$. This is further explained in [Fig. \[fig:forward\_reverse\]]{}, [Fig. \[fig:full-system\]]{} and the list of equations given below.

-   $G_2 : [R_{low},I_{low}] \rightarrow I_{high'}$

-   $G_3 : [R_{low},I_{high'}] \rightarrow S_{high'}$

-   $S_{high'} = G(S_{low}) \approx S_{high}$

-   $F_2 : [R_{high'},I_{high'}] \rightarrow I_{low''}$

-   $F_3 : [R_{high'},I_{low''}] \rightarrow S_{low''}$

-   $S_{low''} = F(S_{high'}) = F(G(S_{low})) \approx S_{low}$

$G_1, G_2, F_1$ and $F_2$ are trainable components (based on U-Net architecture) while $G_3$ and $F_3$ are non-trainable components.

To preserve the cycle consistency of the forward cycle $S_{low}$ and $S_{low''}$ should be the same. When considering the backward cycle, $S_{high}$ and $S_{high''}$ should be the same. Thus, we include a cycle consistency loss by taking the difference between $S_{low}$, $S_{low''}$ and $S_{high}$, $S_{high''}$. as shown in [Eq. ]{}. In addition, using the retinex theory we know that reflectance map in each forward/backward cycle should be equal as well [@deep-retinex-decomp]. Therefore, we include the difference between $R_{low}$, $R_{high'}$ and $R_{high}$, $R_{low'}$ in the cycle consistency loss as well. This loss is given in [Eq. ]{}. Then total cycle consistency loss is given by [Eq. ]{}

$$\label{eq:cycle_loss_s}
    \centering
    \begin{aligned}
    \mathcal{L}_{cyc_S} = 
    &\mathbb{E}_{S_{low}~p(S_{low})} \left[ || F(G(S_{low})) - S_{low} ||_1 \right] +\\
    &\mathbb{E}_{S_{high}~p(S_{high})}[|| G(F(S_{high})) - S_{high} ||_1]
    \end{aligned}$$

$$\label{eq:cycle_loss_r}
    \mathcal{L}_{cyc_R} = || R_{low} - R_{high'} ||_2 + || R_{high} - R_{low'} ||_2$$

$$\label{eq:cycle_loss}
    \mathcal{L}_{cyc} = \mathcal{L}_{cyc_S} +\mathcal{L}_{cyc_R}$$

### Training of enhancement network

The total loss for the enhancement networks G and F including the generator loss is given in [Eq. ]{} and [Eq. ]{} where $H(p,q)$ is the binary cross entropy of distribution $q$ relative to a distribution $p$. $$\label{eq:total_g_loss}
    \begin{aligned}
    \mathcal{L}_G = \mathcal{L}_{cyc} + H(D_{high}(G(S_{low})), \mathbf{1})
    \end{aligned}$$

$$\label{eq:total_f_loss}
    \begin{aligned}
    \mathcal{L}_F = \mathcal{L}_{cyc} +  H(D_{low}(F(S_{high})), \mathbf{1})  
    \end{aligned}$$

These NNs ($G_2, G_3, F_2, F_3$) were trained against a dataset created by adding 485 well-lit images from LOL dataset. The generators were trained to minimize the loss [Eq. ]{} and [Eq. ]{}. We use the Adam optimizer[@adam] at a learning rate of 0.0002 for the discriminator and generator. A decay factor of 0.5 is used. Each model is trained for 500 epochs with batch size 8.

Patched Discriminator
---------------------

![Discriminator model[]{data-label="fig:discriminatormodel"}](figures/discriminator_model){width="\linewidth"}

Usually, GANs use a discriminator to distinguish between real and generated data. When generating images, the whole image is processed and one scalar value between 0 and 1 is predicted. This method is not feasible for this problem because some regions in the image are more specific than the others. For example, an image with low-light in a small area can be identified as a well-light image by the discriminator if the whole image is considered.

Therefore, we use patches from the image and discriminate each patch. Then we take an average of those values. Using this method, we can improve the training of the discriminator. The discriminator model is explained in detail in [Fig. \[fig:discriminatormodel\]]{}. The loss for each discriminator is given in [Eq. ]{} and [Eq. ]{}.

$$\label{eq:total_x_loss}
    \begin{aligned}
    \mathcal{L}_{D_{high}} = H(D_{high}(G(S_{low})), \mathbf{0}) + H(D_{high}(S_{high}), \mathbf{1})
    \end{aligned}$$

$$\label{eq:total_y_loss}
    \begin{aligned}
    \mathcal{L}_{D_{low}} = H(D_{low}(F(S_{high})), \mathbf{0}) + H(D_{low}(S_{low}), \mathbf{1})
    \end{aligned}$$

Summary
-------

The first NN alone can produce $R$,$I$ decomposition for both low light and well-lit images after training. The enhancement network performs the enhancement of the $I$ and puts together $R$ and $I$ to generate enhanced image. The second network is optimized to minimize a wide range of errors.

Results and discussion
======================

Performance metrics
-------------------

In general, the preferred evaluation measure ranking based on visual feedback. For this purpose, this section contains examples of the results generated by the proposed pipeline. The results were also analysed for the pixel-wise mean squared error whenever possible to provide a numerical performance evaluation. Also, we are using the Structural Similarity (SSIM) and Naturalness Image Quality Evaluator (NIQE) as the standard metrics to compare the results. Note that we are using the ratio between the NIQE values of the predicted image and the ground truth image. Furthermore, in order to validate the reasoning behind the specific components of the proposed pipeline, several results were generated to ensure how the inner workings of the system comply with our design goals (instead of treating NN as a black box).

Retinex decomposition
---------------------

Figure \[fig:decomp-result\] shows the performance of the decomposition network for a chosen image. The similarity of $R_{high} \approx R_{low}$ even when $S_{high}$ and $S_{low}$ are different is indicative of the performance of the decomposition network.

![Results of decomposition network. Row 1 (L to R): $S_{low}, R_{low}, I_{low}$. Row 2 (L to R): $S_{high}, R_{high}, I_{high}$[]{data-label="fig:decomp-result"}](figures/decomp_results){width="0.8\linewidth"}

GAN based illumination enhancement
----------------------------------

The GAN is used to enhance the images and illumination maps. The notation for these processes are: $\rightarrow$ denotes NNs trained under adversarial loss and $\rightleftharpoons$ denotes NNs trained under cycle consistency loss (with or without additional loss functions). Two separate experiments were done as per the following approaches. By training the GAN on $[R,I]$, The performance of this network is shown in Figure \[fig:result-cyclegan-illumination\]. It is clearly visible how CycleGAN produced a better illumination enhancement compared to generic GAN.

![Comparison of the results from CycleGAN based illumination enhancement and GAN based illumination enhancement.[]{data-label="fig:result-cyclegan-illumination"}](figures/cycvgan.png){width="0.8\linewidth"}

Complete low-light image enhancement system
-------------------------------------------

Our work studies the results from four approaches to low light enhancement ($G$ and $G_1$ are explained in the proposed work section. $G^a$,$G^b$ are different NNs whose architecture has been kept as close as to $G$ for benchmarking).

1.  $G^a :S_{low} \rightarrow S_{high}$\
    Enhancing low light images using a generic GAN.

2.  $G^b :S_{low} \rightarrow S_{high}$ with $G_1 :(R_{low},I_{Low}) \rightarrow I_{high}$ using supervised training data. Enhancing low light images using Retinexnet.

3.  $G^c :S_{low} \rightarrow S_{high}$ with $G_1 :(R_{low},I_{Low}) \rightarrow I_{high}$ as an intermediate step. Enhancing low light images using a retinex aware GAN pipeline.

4.  $G :S_{low} \rightleftharpoons S_{high}$ with $G_1 :(R_{low},I_{Low}) \rightleftharpoons I_{high}$ as an intermediate step. Enhancing low light images using a retinex aware CycleGAN pipeline.

The results of all the four cases are given in [Fig. \[fig:results-full-pipeline\]]{}. The comparison between them are given in Table. \[tab:numerical\_results\]. As shown in the table both Retinexnet and proposed method have low MSE but when compared with NIQE our method shows better results overall.

![Results from the complete pipeline. Row 1: Low light input images. Row 2: Well-lit ground truth. Row 3: $G^a$ Generic GAN enhanced image. Row 4: $G^b$ Retinexnet enhanced image. Row 5: $G^c$ retinex based GAN enhanced image. Row 5: $G$ Retinex based CycleGAN (ours) enhanced image. []{data-label="fig:results-full-pipeline"}](figures/comparison.png){width="\linewidth"}

        Algorithm         MSE     NIQE ratio     
  --------------------- -------- ------------ -- --
      Ground truth       0.0000     1.0000       
          DCGAN          0.2171     1.4592       
   Retinex based DCGAN   0.0514     1.6967       
       Retinexnet        0.0090     1.7896       
   Proposed algorithm    0.0173     1.7921       

  : Numerical comparision of algorithm performance (MSE - lower is better, NIQE ratio - higher is better)

\[tab:numerical\_results\]

The ablation study of the inner-workings of the model.
------------------------------------------------------

The importance of each component of the model is analysed next. Figures \[fig:gan\_w\_retinex\_result\] and \[fig:cyclegan\_w\_retinex\_result\] show how the NN works in the hidden layers. Layers (from left to right denoting layers from the input to the output) show how the light enhancement process happens sequentially. In figure \[fig:cyclegan\_w\_retinex\_result\] we observe that this results has been further enhanced through cycle consistency loss. In figure \[fig:cyclegan\_w\_retinex\_result\], the similarity of the reflectance of the low light image ($R_{low}$) and well-lit image ($R_{high}'$) highlights the functionality of the cycle consistency loss we introduced. We observe further enhancement in the output $S_{high}'$ which shows that the generative model plays a significant role in image enhancement.

![Caption[]{data-label="fig:my_label"}](figures/retinex-dcgan-layers.png){width=".97\linewidth"}

![Caption[]{data-label="fig:my_label"}](figures/retinex-cycgan-layers.png){width=".97\linewidth"}

Conclusion
==========

Ability to work under low light conditions is an important goal for computer vision. Among other things, enhancing the lighting in images is a crucial milestone in this. Classical image processing algorithms and traditional deep learning algorithms have their strengths and weaknesses.

This paper proposes a pipeline to overcome several weaknesses of traditional deep learning algorithms by being able to use paired and unpaired datasets and use both traditional CNN and GAN architectures in a collaborative manner.

Future work
===========

Even though this paper proposes a methodology to use both paired and unpaired datasets in the enhancement pipeline, the individual components use only one type of dataset. The future work should try to use both types of datasets in each step of the pipeline. The enhanced images of this work shows issues with respect to smoothness. Future work should explore the possibility of enhancing images while preserving natural-like smoothness.

The light enhancement problem is studied in this work as a two-class problem – low light and well-lit. The future work should try to realize different discrete levels of lighting enhancement followed by continuous levels of lighting enhancement. Furthermore, object detection, segmentation, captioning, etc. for low light images could be built upon this work.

Acknowledgment {#acknowledgment .unnumbered}
==============

This research was made possible through the contribution from the citizens of Sri Lanka towards the state-funded university system.

[^1]: LOL contains paired images, but images can be sampled without the need to keep any pair
