I'm aware of the problem of over-fitting. One way to describe it is your Neural Network learning your training data to a high accuracy and performing poorly (generalizing) on new data.

Was wondering if there are situations where having 20 times the number of input neurons on the first hidden layer wouldn't necessarily produce overfitting always.

Here is a screen shot of my work flow. As you can see I'm using test data. The way I've done it is split the data myself using a "Past/Future" column with a P or an F. The future is the last 5% of the sequential time data.

I don't use randomization since I don't think it makes much sense to unsort the sequential time datapoints.

Orange screenshot


  • $\begingroup$ Are you asking how to test this for your problem or how is your presented workflow related to the more general question? $\endgroup$ – oW_ Sep 6 '18 at 22:47
  • $\begingroup$ Mind to share how did you create this graph? I'm trying to find a better way to present machine learning workflow, and this seems quite good. $\endgroup$ – plpopk Sep 7 '18 at 6:19
  • $\begingroup$ @user217471 Orange data analysis app. $\endgroup$ – StudySmarterNotHarder Sep 7 '18 at 15:58

I have not cross-validated my thoughts on overfitting with others (but this answer might be a try).

When we talk about data, we know there are both generalized relationship between input and output, which is what you want your model to learn, and random noises. You can find some plots and a more detailed description here on wikipedia.

In short, being under-fitted means your model have not yet fitted to the real mapping $f(x)\mapsto y$ you intended to teach it. Being over-fitted means your model fitted to much to the random noises in training set, which cannot be generalized to CV set or test set.

Now back to the story of models. There are a bunch of reasons a model can be overfitted, e.g. inappropriate training, wrong choice of model, etc. That being said, on the model side, the prerequisite is quite simple:

A more complex model, in terms of degree of freedom, fitted to a simpler mapping based on a relatively small training set, is more likely to overfit.

A simply linear regression with few features is not likely to overfit, as the degree of freedom, i.e. num of hyper-parameters, is too low.

For neural network, let's say MLP-NN, I can think of two problems with an overlarge hidden layer:

  1. You model is likely to overfit if you don't train it carefully, as the model is quite complicated, while the input has few features, which means the mapping might be relatively simpler.
  2. Though it increases the complexity and computational costs, it might not help with performance. Let's say you applied some technique like drop-out with a large dataset, and successfully trained your model without significant overfitting. However, the real mapping is usually not so complicated on the few features you selected. You might find most of your hidden neurons' behaving nearly identically, and thus can be removed. A better way to account for more complex mapping might be introducing more features.

Figuring out the correct size for hidden layers in NN is a bit of an art, that said I have never used hidden layers with more than twice the number of neurones as the input layer.

As an example the MNIST problem has an input size of 784, I cannot imagine how the network will not overfit if the hidden layer is of size 784*20...

So answering to your question, I think overfit will pretty much occur on those occasions, if unsure, just try it in the MNIST dataser


Your Answer

