{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/37362286","id":37362286,"diff_hunk":"@@ -0,0 +1,506 @@\n+---\n+layout: global\n+title: Decision Trees - SparkML\n+displayTitle: <a href=\"ml-guide.html\">ML</a> - Decision Trees\n+---\n+\n+**Table of Contents**\n+\n+* This will become a table of contents (this text will be scraped).\n+{:toc}\n+\n+\n+# Overview\n+\n+[Decision trees](http://en.wikipedia.org/wiki/Decision_tree_learning)\n+and their ensembles are popular methods for the machine learning tasks of\n+classification and regression. Decision trees are widely used since they are easy to interpret,\n+handle categorical features, extend to the multiclass classification setting, do not require\n+feature scaling, and are able to capture non-linearities and feature interactions. Tree ensemble\n+algorithms such as random forests and boosting are among the top performers for classification and\n+regression tasks.\n+\n+MLlib supports decision trees for binary and multiclass classification and for regression,\n+using both continuous and categorical features. The implementation partitions data by rows,\n+allowing distributed training with millions or even billions of instances.\n+\n+Users can find more information about the decision tree algorithm in the [MLlib Decision Tree guide](mllib-decision-tree.html).  In this section, we demonstrate the Pipelines API for Decision Trees.\n+\n+The Pipelines API for Decision Trees offers a bit more functionality than the original API.  In particular, for classification, users can get the predicted probability of each class (a.k.a. class conditional probabilities).\n+\n+Ensembles of trees (Random Forests and Gradient-Boosted Trees) are described in the [Ensembles guide](ml-ensembles.html).\n+\n+# Inputs and Outputs (Predictions)\n+\n+We list the input and output (prediction) column types here.\n+All output columns are optional; to exclude an output column, set its corresponding Param to an empty string.\n+\n+## Input Columns\n+\n+<table class=\"table\">\n+  <thead>\n+    <tr>\n+      <th align=\"left\">Param name</th>\n+      <th align=\"left\">Type(s)</th>\n+      <th align=\"left\">Default</th>\n+      <th align=\"left\">Description</th>\n+    </tr>\n+  </thead>\n+  <tbody>\n+    <tr>\n+      <td>labelCol</td>\n+      <td>Double</td>\n+      <td>\"label\"</td>\n+      <td>Label to predict</td>\n+    </tr>\n+    <tr>\n+      <td>featuresCol</td>\n+      <td>Vector</td>\n+      <td>\"features\"</td>\n+      <td>Feature vector</td>\n+    </tr>\n+  </tbody>\n+</table>\n+\n+## Output Columns\n+\n+<table class=\"table\">\n+  <thead>\n+    <tr>\n+      <th align=\"left\">Param name</th>\n+      <th align=\"left\">Type(s)</th>\n+      <th align=\"left\">Default</th>\n+      <th align=\"left\">Description</th>\n+      <th align=\"left\">Notes</th>\n+    </tr>\n+  </thead>\n+  <tbody>\n+    <tr>\n+      <td>predictionCol</td>\n+      <td>Double</td>\n+      <td>\"prediction\"</td>\n+      <td>Predicted label</td>\n+      <td></td>\n+    </tr>\n+    <tr>\n+      <td>rawPredictionCol</td>\n+      <td>Vector</td>\n+      <td>\"rawPrediction\"</td>\n+      <td>Vector of length # classes, with the counts of training instance labels at the tree node which makes the prediction</td>\n+      <td>Classification only</td>\n+    </tr>\n+    <tr>\n+      <td>probabilityCol</td>\n+      <td>Vector</td>\n+      <td>\"probability\"</td>\n+      <td>Vector of length # classes equal to rawPrediction normalized to a multinomial distribution</td>\n+      <td>Classification only</td>\n+    </tr>\n+  </tbody>\n+</table>\n+\n+# Examples\n+\n+The below examples demonstrate the Pipelines API for Decision Trees. The main differences between this API and the [original MLlib Decision Tree API](mllib-decision-tree.html) are:\n+\n+* support for ML Pipelines\n+* separation of Decision Trees for classification vs. regression\n+* use of DataFrame metadata to distinguish continuous and categorical features\n+\n+\n+## Classification\n+\n+The following examples load a dataset in LibSVM format, split it into training and test sets, train on the first dataset, and then evaluate on the held-out test set.\n+We use two feature transformers to prepare the data; these help index categories for the label and categorical features, adding metadata to the `DataFrame` which the Decision Tree algorithm can recognize.\n+\n+<div class=\"codetabs\">\n+<div data-lang=\"scala\" markdown=\"1\">\n+\n+More details on parameters can be found in the [Scala API documentation](api/scala/index.html#org.apache.spark.ml.classification.DecisionTreeClassifier).\n+\n+{% highlight scala %}\n+import org.apache.spark.ml.Pipeline\n+import org.apache.spark.ml.classification.DecisionTreeClassifier\n+import org.apache.spark.ml.classification.DecisionTreeClassificationModel\n+import org.apache.spark.ml.feature.{StringIndexer, IndexToString, VectorIndexer}\n+import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator\n+import org.apache.spark.mllib.util.MLUtils\n+\n+// Load and parse the data file, converting it to a DataFrame.\n+val data = MLUtils.loadLibSVMFile(sc, \"data/mllib/sample_libsvm_data.txt\").toDF()\n+\n+// Index labels, adding metadata to the label column.\n+// Fit on whole dataset to include all labels in index.\n+val labelIndexer = new StringIndexer()\n+  .setInputCol(\"label\")\n+  .setOutputCol(\"indexedLabel\")\n+  .fit(data)","path":"docs/ml-decision-tree.md","position":137,"original_position":137,"commit_id":"9dec3d6f5173099cff930af528ee0cfa0ff08729","original_commit_id":"9dec3d6f5173099cff930af528ee0cfa0ff08729","user":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"body":"I agree it's not ideal.  I'd prefer to leave CV out and to fix it in the next release once metadata are available for the prediction column.","created_at":"2015-08-18T22:35:36Z","updated_at":"2015-08-18T22:35:36Z","html_url":"https://github.com/apache/spark/pull/8244#discussion_r37362286","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/8244","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/37362286"},"html":{"href":"https://github.com/apache/spark/pull/8244#discussion_r37362286"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/8244"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8244","id":42615860,"html_url":"https://github.com/apache/spark/pull/8244","diff_url":"https://github.com/apache/spark/pull/8244.diff","patch_url":"https://github.com/apache/spark/pull/8244.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/8244","number":8244,"state":"open","locked":false,"title":"[SPARK-10060] [ML] [DOC] spark.ml DecisionTree user guide","user":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"body":"New user guide section ml-decision-tree.md, including code examples.\r\n\r\nI have run all examples, including the Java ones.\r\n\r\nCC: @manishamde @yanboliang @mengxr ","created_at":"2015-08-17T18:00:28Z","updated_at":"2015-08-18T22:35:36Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6098ff2a06649763d81c1a4107ea7471bf466a73","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/8244/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/8244/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/8244/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/9dec3d6f5173099cff930af528ee0cfa0ff08729","head":{"label":"jkbradley:ml-dt-docs","ref":"ml-dt-docs","sha":"9dec3d6f5173099cff930af528ee0cfa0ff08729","user":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"repo":{"id":19877817,"name":"spark","full_name":"jkbradley/spark","owner":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jkbradley/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/jkbradley/spark","forks_url":"https://api.github.com/repos/jkbradley/spark/forks","keys_url":"https://api.github.com/repos/jkbradley/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jkbradley/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jkbradley/spark/teams","hooks_url":"https://api.github.com/repos/jkbradley/spark/hooks","issue_events_url":"https://api.github.com/repos/jkbradley/spark/issues/events{/number}","events_url":"https://api.github.com/repos/jkbradley/spark/events","assignees_url":"https://api.github.com/repos/jkbradley/spark/assignees{/user}","branches_url":"https://api.github.com/repos/jkbradley/spark/branches{/branch}","tags_url":"https://api.github.com/repos/jkbradley/spark/tags","blobs_url":"https://api.github.com/repos/jkbradley/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jkbradley/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jkbradley/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/jkbradley/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jkbradley/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/jkbradley/spark/languages","stargazers_url":"https://api.github.com/repos/jkbradley/spark/stargazers","contributors_url":"https://api.github.com/repos/jkbradley/spark/contributors","subscribers_url":"https://api.github.com/repos/jkbradley/spark/subscribers","subscription_url":"https://api.github.com/repos/jkbradley/spark/subscription","commits_url":"https://api.github.com/repos/jkbradley/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/jkbradley/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/jkbradley/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/jkbradley/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/jkbradley/spark/contents/{+path}","compare_url":"https://api.github.com/repos/jkbradley/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jkbradley/spark/merges","archive_url":"https://api.github.com/repos/jkbradley/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jkbradley/spark/downloads","issues_url":"https://api.github.com/repos/jkbradley/spark/issues{/number}","pulls_url":"https://api.github.com/repos/jkbradley/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/jkbradley/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/jkbradley/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jkbradley/spark/labels{/name}","releases_url":"https://api.github.com/repos/jkbradley/spark/releases{/id}","created_at":"2014-05-17T03:51:41Z","updated_at":"2015-05-08T20:04:51Z","pushed_at":"2015-08-18T22:22:47Z","git_url":"git://github.com/jkbradley/spark.git","ssh_url":"git@github.com:jkbradley/spark.git","clone_url":"https://github.com/jkbradley/spark.git","svn_url":"https://github.com/jkbradley/spark","homepage":null,"size":122751,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":0,"forks":2,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"f68d024096c90936f9aa4e325141b39f08c72476","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-08-18T20:33:42Z","pushed_at":"2015-08-18T22:30:48Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1647465,"stargazers_count":5126,"watchers_count":5126,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":4676,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":342,"forks":4676,"open_issues":342,"watchers":5126,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/8244"},"html":{"href":"https://github.com/apache/spark/pull/8244"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/8244"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/8244/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/8244/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/8244/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/9dec3d6f5173099cff930af528ee0cfa0ff08729"}}}}