In many implementations/tutorials of GANs that I've seen so far (e.g. this), the generator and discriminator start with no prior knowledge. They continuously improve their performance with training. This makes me wonder — is it possible to use a pre-trained discriminator? I have two motivations for doing so:

  1. Eliminating the overhead of training the discriminator
  2. Being able to use already existing cool models

Would the generator be able to learn just the same, or is it dependent on the fact that they start from scratch?


If you start with perpect discriminator, loss function will be saturated, and gradient of loss will be very small, so feedback for the generator also will be small, and learning will be slow down as a result. Actually, it is allways desired for discriminator and generator to learn balancedly. Additionally, it is claimed that Wasserstein Loss take care of this problem.

You can find more information in this article. (I strongly suggest to read)

Also from the paper " Towards principled methods for training generative adversarial networks":

In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the cost function on $\theta$ better approximates the JSD), and then do gradient steps on $\theta$, alternating these two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates to the generator get consistently worse. The original GAN paper argued that this issue arose from saturation, and switched to another similar cost function that doesn’t have this problem. However, even with this new cost function, updates tend to get worse and optimization gets massively unstable.

Note 1: $\theta$ is parameters of generator

Note 2: JSD: Jensen–Shannon divergence. For optimal discrimantor, loss is equal to $L(D^{ \ast }, g_{\theta})= 2 JSD(P_{r}, P_{g}) - 2log(2)$

Also from the paper of Wasserstein Loss:


  • $\begingroup$ I find this a very good answer. However, I wonder if relativistic GANs wouldn't cope with this and even benefit from the pretrained discriminator. $\endgroup$ – David Dec 24 '20 at 20:45
  • $\begingroup$ @David Thanks for the comment. I honestly heared Relativistic GAN first time from you. Seems interesting paper. $\endgroup$ – Enes Dec 24 '20 at 22:08

Your Answer

