I understand that neural nets are fundamentally interpolative tools. Meaning, given a training dataset, a well trained neural net can approximate values within the domain of the training dataset. However, we are unsure about their behavior once we test against values outside that domain.

Speaking in the context of Imagenet, a NN trained on one of the classes in Imagenet will probably be able to predict an image of the same class outside Imagenet because Imagnet itself covers a huge domain for each class that whatever image we come across in the wild, its features will be accounted for by Imagnet.

Now, this intuition breaks down for me when I talk about simple functions with simple inputs. For example, consider $sin(x)$. Our goal is to train a neural net to predict the function given $x$ with a training domain $[-1, 1]$. Theoretically, the neural net should not be able to predict the values well outside that domain, right? This seems counterintuitive to me because the function behaves in a very simple and periodic way that I find it hard to believe that a neural net cannot figure out the proper transformation of that function even outside the training domain.

In short, are neural nets inherently unable to find a generalizable transformation outside the training domain no matter how simple is the function we are trying to approximate? Is this a property of the Deep Learning framework?

Are there problems where researchers were able to learn a robust generalizable transformation using neural nets outside the training domain? What are the possible conditions so that such results can happen?


The problem you discuss extends past the machine but to the man behind the machine (or woman). ML can be broken down into 3 components, the model, the data, and the learning procedure. This by the way extends to us as well. The model is our brain, the data is our experience and sensory input, and the learning procedure is there but unknown (for now $<$insert evil laugh$>$).

Your model's inability to understand that it is a sin function is normally by construction. First, though lets start with the fact that if someone showed me $sin(x)$ on $[-1,1]$, a sinusoid is the last thing id think of:
enter image description here
It looks almost linear. So for sake of argument i'm going to continue on the assumption you meant an entire period ($[-\pi, \pi])$. enter image description here

Now, given this most people who ever got past pre-calculus would assume sinusoid, as youd expect, but let me show you how unstable that idea is. Lets use a full period, shifted by $\frac{\pi}{2}$ enter image description here
Now i don't know about you but my spidey-sense would make me think of a Gaussian before anything. This is the same sinusoid along an entire period but just the slight difference changes an entire perspective.

Now lets talk about some of the mathematics. Given the uncountable $(x,sin(x))$ pairs between $[-1, 1]$, we can extract the function $sin(x)$ with certainty (assuming infintely differentiable) by simply taking the taylor series along any of the points in that domain. More so, we only need an infinitiscemally small continuous subset of the function to achieve that result, but as a person, would you be able to tell from $[-.0001, .0001]$? (i don't think so!), because by default we quantize, and what were gonna give to the computer is a quantization as well. We are giving it a finite countable set of points and expect it to to generalize to an uncountable set. This is silly, since there exists an infinite set of functions that can fit those exact points (concept of overfitting steps in here).

So if its so unfeasible for a computer to correctly solve the function, why can us humans extrapolate it so well from the $[-\pi, \pi]$ domain? The answer is our preset biases. Growing up we have dealt so much with periodicity, specifically sins and cosines, that graphed period triggered something in our brains and we just know what it is. But this isnt always a benefit, i could draw you a low-ordered polynomial on that domain that would make you think that exact same thought making you wrong.

So going back to my initial point, the problems lies with the models creator and not the model. Like i explained just now, its unfeasible without preset biases to learn the function we want, so what should we do? give up? NO, its our goal as the model architect to figure out what we want and how we can best achieve it! so if we want it to accomplish this task, lets try our best in modeling these biases. A quick example of how you could do that, is to force the function to fit the fourier coefficients, thatll probably solve this quite quickly (ofcourse this limits this models ability to solve for other common functions)

Takeaway: It's not the model thats struggling, its us using preset biases that the model does not have access to. The solution to this is be clever and figure out how to have it either start with these preset biases or learn them.

  • $\begingroup$ It seems like you're saying that the neural network learns a certain function because we decide that we want the NN to learn such function. If that's the case, then I don't think this is answers the question. $\endgroup$ – nbro Jul 22 at 23:08
  • $\begingroup$ @nbro im saying what seems trivial based on our biases isnt trivial when those biases are not present. I think the idea of why alot of NN's have trouble modeling sin(x) in a generalizable mannar (not just overfitting to the domain) is misunderstood because of this exact concept (which is why i do think it is a valid answer). And yes we learn functions that we allow by definition, just genrally we parameterize in a way its difficult for us to understand what that space of fctns looks like. So if we want to accomplish this goal we need to understand how to model our biases $\endgroup$ – mshlis Jul 22 at 23:13
  • $\begingroup$ @mshlis are you trying to say that we should not expect a deep learning model to learn periodic behavior outside its domain unless we provide it with the intuition of periodicity by let's say forcing the Fourier coefficients? Which technically accounts for our bias that periodic behavior is "easy"? $\endgroup$ – Robot0110 Jul 23 at 0:28
  • $\begingroup$ @Robot0110 So that was just one example to show that model design can induce heavy bias! And no, you can learn periodic behavior without making it a constraint! But understand the model will try its hardest to reduce whatever loss you provide, so make sure that the loss your minimizing given your problem is well suited to fit your model the way YOU want $\endgroup$ – mshlis Jul 23 at 1:42

Your Answer

