Why is binary_crossentropy more accurate than categorical_crossentropy for multiclass classification in Keras?
<p>I'm learning how to create convolutional neural networks using Keras. I'm trying to get a high accuracy for the MNIST dataset.</p>

<p>Apparently <code>categorical_crossentropy</code> is for more than 2 classes and <code>binary_crossentropy</code> is for 2 classes. Since there are 10 digits, I should be using <code>categorical_crossentropy</code>. However, after training and testing dozens of models, <code>binary_crossentropy</code> consistently outperforms <code>categorical_crossentropy</code> significantly.</p>

<p>On Kaggle, I got 99+% accuracy using <code>binary_crossentropy</code> and 10 epochs. Meanwhile, I can't get above 97% using <code>categorical_crossentropy</code>, even using 30 epochs (which isn't much, but I don't have a GPU, so training takes forever).</p>

<p>Here's what my model looks like now:</p>



<pre class="lang-python prettyprint-override"><code>model = Sequential()
model.add(Convolution2D(100, 5, 5, border_mode='valid', input_shape=(28, 28, 1), init='glorot_uniform', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(100, 3, 3, init='glorot_uniform', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(100, init='glorot_uniform', activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(100, init='glorot_uniform', activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(10, init='glorot_uniform', activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adamax', metrics=['accuracy'])
</code></pre>