I just ran into "Akaike information criterion", and I noticed this large amount of literature on model selection (also things like BIC seem to exist).

Why don't contemporary machine learning methods take advantage of these BIC and AIC model selection criteria?

  • 9
    $\begingroup$ because nobody's calculating the likelihoods? $\endgroup$ – Aksakal Nov 15 '17 at 14:16
  • 1
    $\begingroup$ What do you mean by "contemporary machine learning methods"? As far as I used AIC and BIC are used frequently. $\endgroup$ – Ferdi Nov 15 '17 at 14:16
  • 4
    $\begingroup$ Also why the -1? Remember there are no stupid questions -- each question tries to shed light on the universe $\endgroup$ – echo Nov 15 '17 at 14:28
  • 4
    $\begingroup$ @echo: I didn't downvote, but I think your question would be improved if you could source/support the main claim (that machine learning methods do take advantage of these BIC and AIC model selection criteria) $\endgroup$ – user603 Nov 15 '17 at 14:32
  • 2
    $\begingroup$ @Aksakal Thanks. I think it is better if questions constructed around a sweeping claim could source that claim. I mean as a general rule. $\endgroup$ – user603 Nov 15 '17 at 14:46

AIC and BIC are used, e.g. in stepwise regression. They are actually part of a larger class of "heuristics", which are also used. For example the DIC (Deviance Information Criterion) is often used in Bayesian Model selection.

However, they are basically "heuristics". While it can be shown, that both the AIC and BIC converge asymptotically towards cross-validation approaches (I think AIC goes towards leave-one-out CV, and BIC towards some other approach, but I am not sure), they are known to under-penalize and over-penalize respectively. I.e. using AIC you will often get a model, which is more complicated than it should be, whereas with BIC you often get a model which is too simplistic.

Since both are related to CV, CV is often a better choice, which does not suffer from these problems.

Then finally there is the issue of the # of parameters which are required for BIC and AIC. With general function approximators (e.g. KNNs) on real-valued inputs, it is possible to "hide" parameters, i.e. to construct a real number which contains the same information as two real numbers (think e.g. of intersecting the digits). In that case, what is the actual number of parameters? On the other hand, with more complicated models, you may have constraints on your parameters, say you can only fit parameters such that $\theta_1 > \theta_2$ (see e.g. here). Or you may have non-identifiability, in which case multiple values of the parameters actually give the same model. In all these case, simply counting of parameters does not give a suitable estimate.

Since many contemporary machine-learning algorithms show these properties (i.e. universal approximation, unclear number of parameters, non-identifiability), AIC and BIC are less useful for these model, than they may seem at first glance.


Some more points that could be clarified:

  1. It seems I was wrong to consider the mapping by interleaving digits a bijection between $\mathbb{R}\rightarrow\mathbb{R}^N$ (see here). However, the details of why this isn't a bijection are a bit hard to understand. However, we don't actually need a bijection for this idea to work (a surjection is enough).
  2. According to proof by Cantor (1877) there must be a bijection between $\mathbb{R}\rightarrow\mathbb{R}^N$. Although this bijection cannot be defined explicitly, it's existence can be proven (but this requires the unproven axiom of choice). This bijection can still be used in a theoretical model (it may not be possible to actually implement this model in a computer), to unpack a single parameter into an arbitrary number of parameters.
  3. We don't actually need the mapping between $\mathbb{R}\rightarrow\mathbb{R}^N$ to be a bijection. Any surjective function $\mathbb{R}\rightarrow\mathbb{R}^N$ is enough to unpack multiple parameters from a single one. Such surjections can be shown to exists as limits to a sequence of other functions (so called space-filling curves, e.g. Peano curve).
  4. Because neither the proof by Cantor is constructive (it simply proves the existence of the bijection without giving an example), nor the space-filling curves (because they only exist as limits of constructive objects and therefore are not constructive themselves), the argument I made is only a theoretical proof. In theory, we could just keep adding parameters to a model to reduce the BIC below any desired value (on the training set). However, in an actual model implementation we have to approximate the space-filling curve, so approximation error may prohibit us from actually doing so (I have not actually tested this).
  5. Because all this requires the axiom of choice, the proof becomes invalid if you don't accept this axiom (although most mathematicians do so). That means, in constructive math this may not be possible, but I don't know what role constructive math plays for statistics.
  6. Identifiability is intrinsically linked to functional complexity. If one simply takes an identifiable $N$-parameter model and adds a superfluous parameter (e.g. not used anywhere), then the new model becomes non-identifiably. Essentially, one is using a model that has the complexity of the $\mathbb{R}^{N+1}$ to solve a problem that has complexity $\mathbb{R}^N$. Similarly, with other forms of non-identifiability. Take for example the case of non-identifiable parameter permutations. In that case, one is using a model that has the complexity of the $\mathbb{R}^N$, however, the actual problem only has the complexity of a set of equivalence classes over the $\mathbb{R}^N$. However, this is only an informal argument, I don't know of any formal treatment of this notion of "complexity".
  • $\begingroup$ Care to chime in on this post stats.stackexchange.com/questions/325129/… ? I've haven't had any luck with it for a while. $\endgroup$ – Reinstate Monica Jan 26 '18 at 21:42
  • 1
    $\begingroup$ @LiKao Can you cite references on the "techniques" of hidding parameters, like the case of intersecting digits. $\endgroup$ – horaceT Sep 1 at 22:16
  • $\begingroup$ @horaceT Unfortunately I don't know of any paper, that gives this example. In papers on MDL there is the notion of "functional complexity" (e.g. lpl.psy.ohio-state.edu/documents/MNP.pdf see eq 10). Often the example is made with constrained parameters (e.g. researchgate.net/publication/…). I like to turn the example around when discussing this, and show that a complex single parameter can capture multiple simple parameters because I find it more intuitive. $\endgroup$ – LiKao Sep 9 at 12:04
  • $\begingroup$ @horaceT Also, if you like a more mathematical treatment, consider that it is proven that filling curves exist, i.e. there is a bijection $f_{1,2}:\mathbb{R} \rightarrow \mathbb{R}^2$. This bijection can easily be used to define bijections $f_{1,N}:\mathbb{R}\rightarrow \mathbb{R}^N$. So for any model with $N$ parameters, I can use $f_{1,N}$ to first get an $N$ dimensional vector from my single parameter, then supply this vector as a parameter to the $N$ parameter model. This gives me a functional equivalent $1$ parameter model. Fitting that model, however would be very complicated at least. $\endgroup$ – LiKao Sep 9 at 12:11
  • $\begingroup$ @LiKao This' quite fascinating. Pls reference said proof of "filing curves". I could see that constrained parameters has "less" degree of freedom. Naively, if f(x,y) = 0, y is just a function of x; you just put g(x) where y is. Can't you do similar things with constrained optimization. $\endgroup$ – horaceT Sep 9 at 21:37

Your Answer

