Unable to train/fine-tune with PReLU in caffe
<p>I am working in face recognition with deep neural network. I am using the <a href="http://www.cbsr.ia.ac.cn/english/CASIA-WebFace-Database.html" rel="nofollow">CASIA-webface</a> database of 10575 classes for training a deep CNN (used by CASIA, see the <a href="http://arxiv.org/abs/1411.7923" rel="nofollow">paper</a> for details) of 10 Convolution, 5 Pooling and 1 Fully Connected layer. For the activation it uses <code>"ReLU"</code> function. I was able to successfully train it using <strong><a href="/questions/tagged/caffe" class="post-tag" title="show questions tagged &#39;caffe&#39;" rel="tag">caffe</a></strong> and obtained the desired performance.</p>

<p>My problem is that, <em>I am unable to train/fine-tune the same CNN using <code>"PReLU"</code> activation</em>. At first, I thought that a simple replace of <code>"ReLU"</code> with <code>"PReLU"</code> will do the job. However, none of <em>fine-tuning</em> (from caffemodel  which was learned with <code>"ReLU"</code>) and <em>learn from scratch</em> strategies worked.</p>

<p>In order to simplify the learning problem, I reduced the training dataset significantly only with 50 classes. However, yet the CNN was unable to learn with <code>"PReLU"</code>, whereas it was able to learn with <code>"ReLU"</code>.</p>

<p>In order to understand that my caffe works fine with <code>"PReLU"</code>, I verified  it by running simple networks (with both <code>"ReLU"</code> and <code>"PReLU"</code>) using cifar10 data and it worked.</p>

<p>I would like to know from the community if anyone has similar observations. Or if anyone can provide any suggestion to overcome this problem.</p>