Tensorflow: GPU memory exhuastion and max pool layers
<p>I have a pretty simple CNN setup (3 convolution layers into a fully connected layer) , using a Geforce GTX 960 GPU with 4GB of memory. I was hitting a GPU resource exhausted issue, and while playing around I discovered if I put a pooling layer (which reduces the overall size of the layer by a factor of 4) it actually made the problem worse. In other words, I was hitting the resource exhausted, and then after removing the pooling layer, I was able to train. I intuitively thought that since a pooling layer would reduce the number of operations downstream during the optimization piece of the training that it would help the memory constraints. Does any know how a max pooling layer could potentially add operations and increase memory like this?  </p>