{"action":"created","issue":{"url":"https://api.github.com/repos/thunder-project/thunder/issues/187","labels_url":"https://api.github.com/repos/thunder-project/thunder/issues/187/labels{/name}","comments_url":"https://api.github.com/repos/thunder-project/thunder/issues/187/comments","events_url":"https://api.github.com/repos/thunder-project/thunder/issues/187/events","html_url":"https://github.com/thunder-project/thunder/pull/187","id":81581424,"number":187,"title":"Linear Regression","user":{"login":"jwittenbach","id":9537932,"avatar_url":"https://avatars.githubusercontent.com/u/9537932?v=3","gravatar_id":"","url":"https://api.github.com/users/jwittenbach","html_url":"https://github.com/jwittenbach","followers_url":"https://api.github.com/users/jwittenbach/followers","following_url":"https://api.github.com/users/jwittenbach/following{/other_user}","gists_url":"https://api.github.com/users/jwittenbach/gists{/gist_id}","starred_url":"https://api.github.com/users/jwittenbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwittenbach/subscriptions","organizations_url":"https://api.github.com/users/jwittenbach/orgs","repos_url":"https://api.github.com/users/jwittenbach/repos","events_url":"https://api.github.com/users/jwittenbach/events{/privacy}","received_events_url":"https://api.github.com/users/jwittenbach/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-27T19:06:35Z","updated_at":"2015-07-19T19:49:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/thunder-project/thunder/pulls/187","html_url":"https://github.com/thunder-project/thunder/pull/187","diff_url":"https://github.com/thunder-project/thunder/pull/187.diff","patch_url":"https://github.com/thunder-project/thunder/pull/187.patch"},"body":"### Overview\r\nThis PR implements not only a couple of new types of linear regression, but also an extensive new framework for all mass regression analyses -- i.e. estimating a set of relationships between a small (local) set of *regressors* and a large (distributed) set of *response variables*.\r\n\r\n### Package Structure\r\nThe ```regression``` folder will be the home for all mass regression analyses. When the relationship between the regressors and the response variables is parametric and linear in the parameters, this is *linear regression* and the corresponding models/algorithms are in the ```linear``` subfolder. When the relationship is parametric and nonlinear in the parameters, such models/algorithms are in the ```nonlinear``` subfolder. One common case of nonlinear regression in neuroscience is the fitting of tuning curves, thus the older ```tuning``` code has been moved to the ```nonlinear``` subdirectory. At the top level of the ```regression``` folder are two pieces of code that can be shared among all regression analyses: a set of ```estimators``` for algorithms that estimate the parameters of parametric regression models, as well as a set of ```transforms``` that manipulate the regressors in common patterns (e.g. centering, z-scoring, etc). \r\n\r\n### Linear Regression\r\n#### UI\r\nThe UI for the linear regression models was carefully laid out and will hopefully serve as a basis for other regression packages as well.\r\n\r\nThe user interacts with these models at the top level through the ```LinearRegression``` factory class. Calling ```LinearRegression(algorithm, *kwargs)```, where ```algorithm``` is a string (e.g ```'ridge'```) will produce a ```LinearRegressionAlgorithm``` of the corresponding type (i.e. subclass of ```LinearRegressionAlgorithm```). Additional keyword arguments for specifying the details of the model are also supplied at this point -- e.g. whether or not to include an intercept term.  The ```LinearRegressionAlgorithm``` itself exposes a ```fit``` method that, when passed a design matrix (containing the samples of the regressors) and a Series object (each record containing the samples of the response variable) returns a fitted model.\r\n\r\nThe fitted model is a ```LinearRegressionModel```. Such a model has properties ```coeffs``` and ```stats``` which return ```Series``` objects that contain the fitted coefficients and the R-squared values from the fit, respectively. ```LinearRegressionModel``` also exposes methods for using the model on new data. The first of these is ```predict``` which takes a new design matrix and returns a ```Series``` object of predicted response variables. The second is ```score``` which takes both a design matrix and a ```Series``` of corresponding responses and returns a ```Series``` of R-squared values indicative of how well the model fits the data. Finally, a courtesy function ```predictAndScore``` performs both of these operations.\r\n\r\n#### API\r\nThe design of the API might also serve as an inspiration for future regression packages.\r\n\r\n```LinearRegressionAlgorithm```: Each ```LinearRegressionAlgorithm``` that derives from the from base class must implement a method with the signature ```def _prepare(self, X)```. This function takes the design matrix as input and returns two objects. First, it must return an ```Estimator``` object which stores all of the information needed to transform the response variables in a single ```Series``` record into an estimate of the coefficients of the linear regression model. This is done through a ```fit``` function on the ```Estimator``` object. An example is the ```PseudoInv``` estimator which computes and stores the pseudoinverse of the design matrix, so that the coefficients can be estimated by simply multiplying this matrix by the response vector. Second, the ```_prepare``` function must also return a ```TransformList```, which is an ordered collection of ```Transform``` objects that are capable of transforming the design matrix prior to any future predictions. Both the ```Estimator``` and ```Transforms``` class definitions are at the top directory in the ```regression``` package, as they should hopefully be useful constructs across all regression analyses. It should be noted that, even if ```intercept=True```, the base class will not make any changes to reflect this, as different algorithms might wish to handle the estimation of the intercept term differently. Thus, each algorithm should make sure to do so. However, the ```TransformList``` will automatically be appended in the base class to take care of ```intercept=True``` and ```zscore=True```, so neither of these need to be handled in the subclass. Each ```LinearRegressionAlgorithm``` must also implement its own initialization method, ```def __init__(self, *kwargs)```. The keyword arguments ```intercept``` and ```zscore``` are handled by the base class ```__init__``` function, so this should always be called in the subclass's own ```__init__```. This is where additional parameters that are specific to the algorithm are handled (e.g. regularization strength for ridge regression).\r\n\r\n```LinearRegressionModel```: The ```LinearRegressionModel``` is an object that stores the model itself. It is backed by an RDD of ```LocalRegressionModels```. These local models store the regression coefficients and have methods for producing predictions and R-squared statistics when given a design matrix of the the correct form. Thus the ```LinearRegressionModel``` also contains a ```TransformList``` that indicates the transforms that should first be applied to the design matrix before it is shipped off to the ```LocalRegressionModels``` for prediction/scoring.\r\n\r\n\r\n#### Algorithms\r\nWith this initial update, we have implemented four different linear regression algorithms:\r\n\r\n```OrdinaryLinearRegression```: Finds the coefficients by minimizing the ordinary least-squares loss function via the pseudoinverse of the design matrix.\r\n\r\n```TikhonovLinearRegression```: Regularized regression that fits the coefficients by minimizing the the least-squares loss function with a penalty for the L2 norm of a vector of user-determined linear combinations of the coefficients. Because of the nice properties of the L2 norm, this problem can be recast as an equivalent augmented ordinary least-squares problem.\r\n\r\n```RidgeLinearRegression```: A special case of ```TikhonovLinearRegression``` where the matrix specifying the linear combinations is the identity matrix, thus penalizing the L2 norm of the vector of coefficients.\r\n\r\n```ConstrainedLinearRegression```: Finds the coefficients that minimize the ordinary least-squares loss function under a set of linear inequality constraints on the coefficients. [edit: removed for future PR due to complexity of adding CVXOPT package (which must be complied) as a dependency]\r\n\r\n### Future Directions\r\n* Additional linear regression algorithms can be added. Examples include lasso regression and basis function expansions on regressors\r\n* Additional regression packages can be added. Examples include cascade models and nonparametric tuning curve models using basis functions."},"comment":{"url":"https://api.github.com/repos/thunder-project/thunder/issues/comments/122697887","html_url":"https://github.com/thunder-project/thunder/pull/187#issuecomment-122697887","issue_url":"https://api.github.com/repos/thunder-project/thunder/issues/187","id":122697887,"user":{"login":"freeman-lab","id":3387500,"avatar_url":"https://avatars.githubusercontent.com/u/3387500?v=3","gravatar_id":"","url":"https://api.github.com/users/freeman-lab","html_url":"https://github.com/freeman-lab","followers_url":"https://api.github.com/users/freeman-lab/followers","following_url":"https://api.github.com/users/freeman-lab/following{/other_user}","gists_url":"https://api.github.com/users/freeman-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/freeman-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freeman-lab/subscriptions","organizations_url":"https://api.github.com/users/freeman-lab/orgs","repos_url":"https://api.github.com/users/freeman-lab/repos","events_url":"https://api.github.com/users/freeman-lab/events{/privacy}","received_events_url":"https://api.github.com/users/freeman-lab/received_events","type":"User","site_admin":false},"created_at":"2015-07-19T19:49:13Z","updated_at":"2015-07-19T19:49:13Z","body":"Fixed conflicts and merged into master https://github.com/thunder-project/thunder/commit/490d3f92e793df5bc847fcf72687f7bde47f05fe"}}