{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6039","labels_url":"https://api.github.com/repos/apache/spark/issues/6039/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6039/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6039/events","html_url":"https://github.com/apache/spark/pull/6039","id":74872824,"number":6039,"title":"[Spark-7514][MLlib] Add MinMaxScaler to feature transformation","user":{"login":"hhbyyh","id":7981698,"avatar_url":"https://avatars.githubusercontent.com/u/7981698?v=3","gravatar_id":"","url":"https://api.github.com/users/hhbyyh","html_url":"https://github.com/hhbyyh","followers_url":"https://api.github.com/users/hhbyyh/followers","following_url":"https://api.github.com/users/hhbyyh/following{/other_user}","gists_url":"https://api.github.com/users/hhbyyh/gists{/gist_id}","starred_url":"https://api.github.com/users/hhbyyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hhbyyh/subscriptions","organizations_url":"https://api.github.com/users/hhbyyh/orgs","repos_url":"https://api.github.com/users/hhbyyh/repos","events_url":"https://api.github.com/users/hhbyyh/events{/privacy}","received_events_url":"https://api.github.com/users/hhbyyh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-05-10T09:00:12Z","updated_at":"2015-05-12T19:39:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6039","html_url":"https://github.com/apache/spark/pull/6039","diff_url":"https://github.com/apache/spark/pull/6039.diff","patch_url":"https://github.com/apache/spark/pull/6039.patch"},"body":"jira: https://issues.apache.org/jira/browse/SPARK-7514\r\nAdd a popular scaling method to feature component, which is commonly known as min-max normalization or Rescaling.\r\n\r\nCore function is,\r\nNormalized(x) = (x - min) / (max - min) * scale + newBase\r\n\r\nwhere `newBase` and `scale` are parameters (type Double) of the `VectorTransformer`. `newBase` is the new minimum number for the features, and `scale` controls the ranges after transformation. This is a little complicated than the basic MinMax normalization, yet it provides flexibility so that users can control the range more specifically. like [0.1, 0.9] in some NN application.\r\n\r\nFor case that `max == min`, 0.5 is used as the raw value. (0.5 * scale + newBase)\r\nI'll add UT once the design got settled ( and this is not considered as too naive)\r\n\r\nreference:\r\n http://en.wikipedia.org/wiki/Feature_scaling\r\nhttp://stn.spotfire.com/spotfire_client_help/index.htm#norm/norm_scale_between_0_and_1.htm"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/101396078","html_url":"https://github.com/apache/spark/pull/6039#issuecomment-101396078","issue_url":"https://api.github.com/repos/apache/spark/issues/6039","id":101396078,"user":{"login":"mengxr","id":829644,"avatar_url":"https://avatars.githubusercontent.com/u/829644?v=3","gravatar_id":"","url":"https://api.github.com/users/mengxr","html_url":"https://github.com/mengxr","followers_url":"https://api.github.com/users/mengxr/followers","following_url":"https://api.github.com/users/mengxr/following{/other_user}","gists_url":"https://api.github.com/users/mengxr/gists{/gist_id}","starred_url":"https://api.github.com/users/mengxr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mengxr/subscriptions","organizations_url":"https://api.github.com/users/mengxr/orgs","repos_url":"https://api.github.com/users/mengxr/repos","events_url":"https://api.github.com/users/mengxr/events{/privacy}","received_events_url":"https://api.github.com/users/mengxr/received_events","type":"User","site_admin":false},"created_at":"2015-05-12T19:39:18Z","updated_at":"2015-05-12T19:39:18Z","body":"@hhbyyh The pipeline API is a better place for feature transformers. It would be nice if you can implement `MinMaxScaler` under `spark.ml` using the pipeline API. Instead of `scale` and `newBase`, it would be easier for users to specify `min` and `max` after scaling, which is similar to what sklearn has."}}