{"action":"created","issue":{"url":"https://api.github.com/repos/Evizero/LearnBase.jl/issues/7","labels_url":"https://api.github.com/repos/Evizero/LearnBase.jl/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/Evizero/LearnBase.jl/issues/7/comments","events_url":"https://api.github.com/repos/Evizero/LearnBase.jl/issues/7/events","html_url":"https://github.com/Evizero/LearnBase.jl/issues/7","id":114644451,"number":7,"title":"Overlap","user":{"login":"tbreloff","id":933338,"avatar_url":"https://avatars.githubusercontent.com/u/933338?v=3","gravatar_id":"","url":"https://api.github.com/users/tbreloff","html_url":"https://github.com/tbreloff","followers_url":"https://api.github.com/users/tbreloff/followers","following_url":"https://api.github.com/users/tbreloff/following{/other_user}","gists_url":"https://api.github.com/users/tbreloff/gists{/gist_id}","starred_url":"https://api.github.com/users/tbreloff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbreloff/subscriptions","organizations_url":"https://api.github.com/users/tbreloff/orgs","repos_url":"https://api.github.com/users/tbreloff/repos","events_url":"https://api.github.com/users/tbreloff/events{/privacy}","received_events_url":"https://api.github.com/users/tbreloff/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-02T17:32:56Z","updated_at":"2015-11-02T20:02:51Z","closed_at":null,"body":"I'm starting this as a catch-all for discussion about concepts that heavily overlap between disciplines.  I had started to contemplate how to merge the implementation that you're developing here with the concepts/requirements that I see in OnlineStats/OnlineAI.\r\n\r\n(sorry... this is a little rambling...)\r\n\r\nSimilar:\r\n- Link functions in GLM (https://en.wikipedia.org/wiki/Generalized_linear_model#Link_function)\r\n- Activation functions in ANNs (https://en.wikipedia.org/wiki/Activation_function)\r\n- Kernels (could be considered similar with the right perspective)\r\n\r\nA link function is effectively of the form:  `f(y) = x'w + b` or `y = f'(x'w + b)`\r\nAn activation function is effectively:  `y = f(x'w + b)`... the name is more a holdover from the perceptron's discontinuous {0,1} classification, but over time has seemed to blend more with the concept of a link function.  So we see a \"sigmoid activation\" is effectively the same as the \"inverse logit link\" in the GLM sense.  It's valuable to think of a neural net with no hidden layers to be conceptually similar to a GLM... we are transforming the inner product of inputs and weights (plus bias), with the goal of maximizing the likelihood that w/b generate the relationship.\r\n\r\nNow, neural nets are built from connected layers of these GLMs, which just happen to have solvers that exploit the known structure to provide better \"measures of fit\" for the earlier layers.  The likelihood is in relation to the whole of the structure, as opposed to just a single function of `f(x'w + b)`.  \r\n\r\nLets jump ahead and think of how we can apply a \"layering\" concept to other areas.  You mentioned in another issue that you would like to do something like:\r\n```\r\ncs = fit(CenterScale(), Xtrain)\r\nXtrain2 = predict(cs, Xtrain)\r\n\r\npca = fit(PCA(), Xtrain2)\r\nXtrain3 = predict(pca, Xtrain2)\r\n\r\nmyfit = fit(MyModel(), Xtrain3)\r\nyhat = predict(myfit, Xtrain3)\r\n```\r\n\r\nIn some sense this is an extremely specific 3-layer neural net, and I think some abstractions can be seen.  Assume we have 3 input variables, and 1 target.  Imagine instead:\r\n- the first layer is `y1 = f1(x'w1 + b1)`, with w1 the identity matrix, and f1 the identity function.\r\n- the second layer is `y2 = f2(y1'w2 + b2)`, with w2 as the principal components loading matrix, f2 the identity function\r\n- the third later is the model `y3 = f3(y2, w3, b3)`\r\n\r\nThinking about things like this I think will open your mind to some alternative ways to think about modeling pipelines... specifically why are you focused on de-meaning the inputs, or reducing dimensions using PCA, if your final goal is really to have some larger function which maps initial inputs to final target values.  The concept behind [PLS](http://www.ats.ucla.edu/stat/sas/library/pls.pdf) is very similar... instead of 2 independent steps of \"reduce dimension, then fit\", it combines into a unified step \"reduce dimensions and fit\".  This is how neural nets work as well, and backprop just gives a method to fit the weights of earlier models using the chain rule and intermediate gradients.\r\n\r\nThere are also parallels to a lot of the deep learning approaches... namely the use of Restricted Boltzmann Machines and Stacked Autoencoders (both have heavy parallels with PCA and dimensionality reduction) as inputs into a simpler predictive model.  There has been good success in using these models/algorithms simply as weight initialization in a larger system, where those models are effectively re-fit using the final target as driving the weight updates, here with a heavy parallel to PLS.\r\n\r\nThe point I'm trying to get across with all these tangents is that I think it can be very helpful when designing, to think about how simpler models (pca, svm, glm, ...) could be reinterpreted in terms of a more complicated framework.  With the right abstractions, we could recreate something like PLS automatically by plugging the generalized PCA with generalized logistic regression. "},"comment":{"url":"https://api.github.com/repos/Evizero/LearnBase.jl/issues/comments/153141248","html_url":"https://github.com/Evizero/LearnBase.jl/issues/7#issuecomment-153141248","issue_url":"https://api.github.com/repos/Evizero/LearnBase.jl/issues/7","id":153141248,"user":{"login":"Evizero","id":10854026,"avatar_url":"https://avatars.githubusercontent.com/u/10854026?v=3","gravatar_id":"","url":"https://api.github.com/users/Evizero","html_url":"https://github.com/Evizero","followers_url":"https://api.github.com/users/Evizero/followers","following_url":"https://api.github.com/users/Evizero/following{/other_user}","gists_url":"https://api.github.com/users/Evizero/gists{/gist_id}","starred_url":"https://api.github.com/users/Evizero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Evizero/subscriptions","organizations_url":"https://api.github.com/users/Evizero/orgs","repos_url":"https://api.github.com/users/Evizero/repos","events_url":"https://api.github.com/users/Evizero/events{/privacy}","received_events_url":"https://api.github.com/users/Evizero/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T20:02:51Z","updated_at":"2015-11-02T20:02:51Z","body":"I agree with pretty much everything expect the following\r\n\r\n> For example, we don't need this when we are going to define this explicitly later. In my mind, it's just adding to the codebase and complexity without much benefit. \r\n\r\nI very strongly disagree (aside from the stubs that have the \"not implemented\" macro). The MarginBased vs DistanceBased loss is the most useful aspect of the Loss functions to me. I absolutely need those in some form for KSVM. Most importantly they allow dispatch on Classification vs Regression on the loss level. Also I don't want the loss functions to be some invisible backbone stuff that shouldn't be touched by endusers. I would like them to be useful on their own for course exercises and experiments. I also need the properties. For example the Pegasos algorithm for SVMs needs the loss to be lipschitzcontinuous.\r\n\r\n> ... Defines loss function and derivatives (but not gradients?) \r\n\r\nWell, practically speaking the difference is the type of the parameters. might as well use one verb for both. That being said, I have no strong opinion on this.\r\n\r\n> I actually go back and forth on whether this should be part of Cost... i.e. there could be a Cost which is the sum of a Loss and a Penalty\r\n\r\nI'm also not completely set on this. Initially loss+penalty was the cost to me, but that would be inconsistent with the loss theory. I'll soon sketch out deterministic empirical risk minimization using Optim.jl and see where it takes me\r\n\r\n>  I forked the repo, and I'll start to build up some of the building blocks using this concept and see how far I get.\r\n\r\nsweet! I'll continue working on loss/predictor as I progress on KSVM"}}