How to implement weight decay in tensorflow as in Caffe
<p>In Caffe we have a decay_ratio which is usually set as 0.0005. Then all trainable parameters, e.g., W matrix in FC6 will be decayed by:
W = W * (1 - 0.0005)
after we applied the gradient to it.</p>

<p>I go through many tutorial tensorflow codes, but do not see how people implement this weight decay to prevent numerical problems (very large absolute values)</p>

<p>I my experiences, I often run into numerical problems aften 100k iterations during training.</p>

<p>I also go through related questions at stackoverflow, e.g., 
<a href="https://stackoverflow.com/questions/34986911/how-to-set-weight-cost-strength-in-tensorflow">How to set weight cost strength in TensorFlow?</a>
However, the solution seems a little different as implemented in Caffe.</p>

<p>Does anyone has similar concerns? Thank you.</p>