How to check relu gradient
<p>I'm trying to implement ANN and I also wrote a numerical gradient check for backpropagation.
When I use sigmoid function, the numerical gradient check works correctly.<br>
However, when I use relu activation, the gradient check fails.</p>

<p>The gradient I get is written as follows:</p>

<pre><code>switch opts.act_function
    case 'relu'
        d_act = a{i} &gt; 0;
    case 'sigmoid'
        d_act = a{i} * (1 - a{i});
end
</code></pre>

<p>My question is that there is no gradient at 0. If I set the subgradient of 0 to 0, is this correct?</p>