{"action":"created","comment":{"url":"https://api.github.com/repos/apache/flink/pulls/comments/31896759","id":31896759,"diff_hunk":"@@ -24,4 +24,198 @@ under the License.\n * This will be replaced by the TOC\n {:toc}\n \n-Coming soon.\n+## Introduction\n+\n+FlinkML is designed to make learning from your data a straight-forward process, abstracting away\n+the complexities that usually come with having to deal with big data learning tasks. In this\n+quick-start guide we will show just how easy it is to solve a simple supervised learning problem\n+using FlinkML. But first some basics, feel free to skip the next few lines if you're already\n+familiar with Machine Learning (ML)\n+\n+As defined by Murphy [cite ML-APP] ML deals with detecting patterns in data, and using those\n+learned patterns to make predictions about the future. We can categorize most ML algorithms into\n+two major categories: Supervised and Unsupervised Learning.\n+\n+* Supervised Learning deals with learning a function (mapping) from a set of inputs\n+(predictors) to a set of outputs. The learning is done using a __training set__ of (input,\n+output) pairs that we use to approximate the mapping function. Supervised learning problems are\n+further divided into classification and regression problems. In classification problems we try to\n+predict the __class__ that an example belongs to, for example whether a user is going to click on\n+an ad or not. Regression problems are about predicting (real) numerical values,  often called the dependent\n+variable, for example what the temperature will be tomorrow.\n+\n+* Unsupervised learning deals with discovering patterns and regularities in the data. An example\n+of this would be __clustering__, where we try to discover groupings of the data from the\n+descriptive features. Unsupervised learning can also be used for feature selection, for example\n+through [principal components analysis](https://en.wikipedia.org/wiki/Principal_component_analysis).\n+\n+## Loading data\n+\n+For loading data to be used with FlinkML we can use the ETL capabilities of Flink, or specialized\n+functions for formatted data, such as the LibSVM format. For supervised learning problems it is\n+common to use the `LabeledVector` class to represent the `(features, label)` examples. A `LabeledVector`\n+object will have a FlinkML `Vector` member representing the features of the example and a `Double`\n+member which represents the label, which could be the class in a classification problem, or the dependent\n+variable for a regression problem.\n+\n+# TODO: Get dataset that has separate train and test sets\n+As an example, we can use the Breast Cancer Wisconsin (Diagnostic) Data Set, which you can\n+[download from the UCI ML repository](http://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data).\n+\n+We can load the data as a `DataSet[String]` first:\n+\n+{% highlight scala %}\n+\n+val cancer = env.readCsvFile[(String, String, String, String, String, String, String, String, String, String, String)](\"/path/to/breast-cancer-wisconsin.data\")\n+\n+{% endhighlight %}\n+\n+The dataset has some missing values indicated by `?`. We can filter those rows out and\n+then transform the data into a `DataSet[LabeledVector]`. This will allow us to use the\n+dataset with the FlinkML classification algorithms.\n+\n+{% highlight scala %}\n+\n+val cancerLV = cancer\n+  .map(_.productIterator.toList)\n+  .filter(!_.contains(\"?\"))\n+  .map{list =>\n+    val numList = list.map(_.asInstanceOf[String].toDouble)\n+    LabeledVector(numList(11), DenseVector(numList.take(10).toArray))\n+    }\n+\n+{% endhighlight %}\n+\n+We can then use this data to train a learner.\n+\n+A common format for ML datasets is the LibSVM format and a number of datasets using that format can be\n+found [in the LibSVM datasets website](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/). FlinkML provides utilities for loading\n+datasets using the LibSVM format through the `readLibSVM` function available through the MLUtils object.\n+You can also save datasets in the LibSVM format using the `writeLibSVM` function.\n+Let's import the Adult (a9a) dataset. You can download the \n+[training set here](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/a8a)\n+and the [test set here](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/a8a.t).\n+\n+We can simply import the dataset then using:\n+\n+{% highlight scala %}\n+\n+val adultTrain = MLUtils.readLibSVM(\"path/to/a8a\")\n+val adultTest = MLUtils.readLibSVM(\"path/to/a8a.t\")\n+\n+{% endhighlight %}\n+\n+This gives us a `DataSet[LabeledVector]` that we will use in the following section to create a classifier.\n+\n+Due to an error in the test dataset we have to adjust the test data using the following code, to \n+ensure that the dimensionality of all test examples is 123, as with the training set:\n+\n+{% highlight scala %}\n+\n+val adjustedTest = adultTest.map{lv =>\n+      val vec = lv.vector.asBreeze\n+      val padded = vec.padTo(123, 0.0).toDenseVector\n+      val fvec = padded.fromBreeze\n+      LabeledVector(lv.label, fvec)\n+    }\n+\n+{% endhighlight %}\n+\n+## Classification\n+\n+Once we have imported the dataset we can train a `Predictor` such as a linear SVM classifier.\n+\n+{% highlight scala %}\n+\n+val svm = SVM()\n+  .setBlocks(env.getParallelism)\n+  .setIterations(100)\n+  .setRegularization(0.001)\n+  .setStepsize(0.1)\n+  .setSeed(42)\n+\n+svm.fit(adultTrain)\n+\n+{% endhighlight %}\n+\n+Let's now make predictions on the test set and see how well we do in terms of absolute error\n+We will also create a function that thresholds the predictions to the {-1, 1} scale that the\n+dataset uses.\n+\n+{% highlight scala %}\n+\n+def thresholdPredictions(predictions: DataSet[(Double, Double)])\n+: DataSet[(Double, Double)] = {\n+  predictions.map {\n+    truthPrediction =>\n+      val truth = truthPrediction._1\n+      val prediction = truthPrediction._2\n+      val thresholdedPrediction = if (prediction > 0.0) 1.0 else -1.0\n+      (truth, thresholdedPrediction)\n+  }\n+}\n+\n+val predictionPairs = thresholdPredictions(svm.predict(adjustedTest))\n+\n+val absoluteErrorSum = predictionPairs.collect().map{\n+  case (truth, prediction) => Math.abs(truth - prediction)}.sum","path":"docs/libs/ml/quickstart.md","position":139,"original_position":139,"commit_id":"923fe52cd0998e668e2bc27aeff27e8067eaddab","original_commit_id":"923fe52cd0998e668e2bc27aeff27e8067eaddab","user":{"login":"tillrohrmann","id":5756858,"avatar_url":"https://avatars.githubusercontent.com/u/5756858?v=3","gravatar_id":"","url":"https://api.github.com/users/tillrohrmann","html_url":"https://github.com/tillrohrmann","followers_url":"https://api.github.com/users/tillrohrmann/followers","following_url":"https://api.github.com/users/tillrohrmann/following{/other_user}","gists_url":"https://api.github.com/users/tillrohrmann/gists{/gist_id}","starred_url":"https://api.github.com/users/tillrohrmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tillrohrmann/subscriptions","organizations_url":"https://api.github.com/users/tillrohrmann/orgs","repos_url":"https://api.github.com/users/tillrohrmann/repos","events_url":"https://api.github.com/users/tillrohrmann/events{/privacy}","received_events_url":"https://api.github.com/users/tillrohrmann/received_events","type":"User","site_admin":false},"body":"Maybe we can also print the results/error here.","created_at":"2015-06-08T09:03:23Z","updated_at":"2015-06-08T09:03:23Z","html_url":"https://github.com/apache/flink/pull/792#discussion_r31896759","pull_request_url":"https://api.github.com/repos/apache/flink/pulls/792","_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/comments/31896759"},"html":{"href":"https://github.com/apache/flink/pull/792#discussion_r31896759"},"pull_request":{"href":"https://api.github.com/repos/apache/flink/pulls/792"}}},"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/792","id":37013683,"html_url":"https://github.com/apache/flink/pull/792","diff_url":"https://github.com/apache/flink/pull/792.diff","patch_url":"https://github.com/apache/flink/pull/792.patch","issue_url":"https://api.github.com/repos/apache/flink/issues/792","number":792,"state":"open","locked":false,"title":"[FLINK-2072] [ml]  [docs] Add a quickstart guide for FlinkML","user":{"login":"thvasilo","id":9048995,"avatar_url":"https://avatars.githubusercontent.com/u/9048995?v=3","gravatar_id":"","url":"https://api.github.com/users/thvasilo","html_url":"https://github.com/thvasilo","followers_url":"https://api.github.com/users/thvasilo/followers","following_url":"https://api.github.com/users/thvasilo/following{/other_user}","gists_url":"https://api.github.com/users/thvasilo/gists{/gist_id}","starred_url":"https://api.github.com/users/thvasilo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thvasilo/subscriptions","organizations_url":"https://api.github.com/users/thvasilo/orgs","repos_url":"https://api.github.com/users/thvasilo/repos","events_url":"https://api.github.com/users/thvasilo/events{/privacy}","received_events_url":"https://api.github.com/users/thvasilo/received_events","type":"User","site_admin":false},"body":"This is an initial version of the quickstart guide. There are some issues that still need to be addressed such as the validity of standardizing the data, and whether the complete code example should be included in an examples package for FlinkML.","created_at":"2015-06-05T09:14:25Z","updated_at":"2015-06-08T09:03:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":"abd424c295fdad772fd9956b3fe1e77f3617e8ff","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/flink/pulls/792/commits","review_comments_url":"https://api.github.com/repos/apache/flink/pulls/792/comments","review_comment_url":"https://api.github.com/repos/apache/flink/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/flink/issues/792/comments","statuses_url":"https://api.github.com/repos/apache/flink/statuses/923fe52cd0998e668e2bc27aeff27e8067eaddab","head":{"label":"thvasilo:quickstart-ml","ref":"quickstart-ml","sha":"923fe52cd0998e668e2bc27aeff27e8067eaddab","user":{"login":"thvasilo","id":9048995,"avatar_url":"https://avatars.githubusercontent.com/u/9048995?v=3","gravatar_id":"","url":"https://api.github.com/users/thvasilo","html_url":"https://github.com/thvasilo","followers_url":"https://api.github.com/users/thvasilo/followers","following_url":"https://api.github.com/users/thvasilo/following{/other_user}","gists_url":"https://api.github.com/users/thvasilo/gists{/gist_id}","starred_url":"https://api.github.com/users/thvasilo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thvasilo/subscriptions","organizations_url":"https://api.github.com/users/thvasilo/orgs","repos_url":"https://api.github.com/users/thvasilo/repos","events_url":"https://api.github.com/users/thvasilo/events{/privacy}","received_events_url":"https://api.github.com/users/thvasilo/received_events","type":"User","site_admin":false},"repo":{"id":31908956,"name":"flink","full_name":"thvasilo/flink","owner":{"login":"thvasilo","id":9048995,"avatar_url":"https://avatars.githubusercontent.com/u/9048995?v=3","gravatar_id":"","url":"https://api.github.com/users/thvasilo","html_url":"https://github.com/thvasilo","followers_url":"https://api.github.com/users/thvasilo/followers","following_url":"https://api.github.com/users/thvasilo/following{/other_user}","gists_url":"https://api.github.com/users/thvasilo/gists{/gist_id}","starred_url":"https://api.github.com/users/thvasilo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thvasilo/subscriptions","organizations_url":"https://api.github.com/users/thvasilo/orgs","repos_url":"https://api.github.com/users/thvasilo/repos","events_url":"https://api.github.com/users/thvasilo/events{/privacy}","received_events_url":"https://api.github.com/users/thvasilo/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/thvasilo/flink","description":"Mirror of Apache Flink","fork":true,"url":"https://api.github.com/repos/thvasilo/flink","forks_url":"https://api.github.com/repos/thvasilo/flink/forks","keys_url":"https://api.github.com/repos/thvasilo/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/thvasilo/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/thvasilo/flink/teams","hooks_url":"https://api.github.com/repos/thvasilo/flink/hooks","issue_events_url":"https://api.github.com/repos/thvasilo/flink/issues/events{/number}","events_url":"https://api.github.com/repos/thvasilo/flink/events","assignees_url":"https://api.github.com/repos/thvasilo/flink/assignees{/user}","branches_url":"https://api.github.com/repos/thvasilo/flink/branches{/branch}","tags_url":"https://api.github.com/repos/thvasilo/flink/tags","blobs_url":"https://api.github.com/repos/thvasilo/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/thvasilo/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/thvasilo/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/thvasilo/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/thvasilo/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/thvasilo/flink/languages","stargazers_url":"https://api.github.com/repos/thvasilo/flink/stargazers","contributors_url":"https://api.github.com/repos/thvasilo/flink/contributors","subscribers_url":"https://api.github.com/repos/thvasilo/flink/subscribers","subscription_url":"https://api.github.com/repos/thvasilo/flink/subscription","commits_url":"https://api.github.com/repos/thvasilo/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/thvasilo/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/thvasilo/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/thvasilo/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/thvasilo/flink/contents/{+path}","compare_url":"https://api.github.com/repos/thvasilo/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/thvasilo/flink/merges","archive_url":"https://api.github.com/repos/thvasilo/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/thvasilo/flink/downloads","issues_url":"https://api.github.com/repos/thvasilo/flink/issues{/number}","pulls_url":"https://api.github.com/repos/thvasilo/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/thvasilo/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/thvasilo/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/thvasilo/flink/labels{/name}","releases_url":"https://api.github.com/repos/thvasilo/flink/releases{/id}","created_at":"2015-03-09T16:14:55Z","updated_at":"2015-04-24T12:16:55Z","pushed_at":"2015-06-05T09:37:50Z","git_url":"git://github.com/thvasilo/flink.git","ssh_url":"git@github.com:thvasilo/flink.git","clone_url":"https://github.com/thvasilo/flink.git","svn_url":"https://github.com/thvasilo/flink","homepage":null,"size":49820,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"92d3251261808f72bb5ef390ac1cf9b396f1d185","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20587599,"name":"flink","full_name":"apache/flink","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/flink","description":"Mirror of Apache Flink","fork":false,"url":"https://api.github.com/repos/apache/flink","forks_url":"https://api.github.com/repos/apache/flink/forks","keys_url":"https://api.github.com/repos/apache/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/flink/teams","hooks_url":"https://api.github.com/repos/apache/flink/hooks","issue_events_url":"https://api.github.com/repos/apache/flink/issues/events{/number}","events_url":"https://api.github.com/repos/apache/flink/events","assignees_url":"https://api.github.com/repos/apache/flink/assignees{/user}","branches_url":"https://api.github.com/repos/apache/flink/branches{/branch}","tags_url":"https://api.github.com/repos/apache/flink/tags","blobs_url":"https://api.github.com/repos/apache/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/flink/languages","stargazers_url":"https://api.github.com/repos/apache/flink/stargazers","contributors_url":"https://api.github.com/repos/apache/flink/contributors","subscribers_url":"https://api.github.com/repos/apache/flink/subscribers","subscription_url":"https://api.github.com/repos/apache/flink/subscription","commits_url":"https://api.github.com/repos/apache/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/flink/contents/{+path}","compare_url":"https://api.github.com/repos/apache/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/flink/merges","archive_url":"https://api.github.com/repos/apache/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/flink/downloads","issues_url":"https://api.github.com/repos/apache/flink/issues{/number}","pulls_url":"https://api.github.com/repos/apache/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/flink/labels{/name}","releases_url":"https://api.github.com/repos/apache/flink/releases{/id}","created_at":"2014-06-07T07:00:10Z","updated_at":"2015-06-08T07:23:13Z","pushed_at":"2015-06-08T08:47:30Z","git_url":"git://github.com/apache/flink.git","ssh_url":"git@github.com:apache/flink.git","clone_url":"https://github.com/apache/flink.git","svn_url":"https://github.com/apache/flink","homepage":null,"size":2349581,"stargazers_count":444,"watchers_count":444,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":282,"mirror_url":"git://git.apache.org/incubator-flink.git","open_issues_count":54,"forks":282,"open_issues":54,"watchers":444,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/792"},"html":{"href":"https://github.com/apache/flink/pull/792"},"issue":{"href":"https://api.github.com/repos/apache/flink/issues/792"},"comments":{"href":"https://api.github.com/repos/apache/flink/issues/792/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/flink/pulls/792/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/flink/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/flink/pulls/792/commits"},"statuses":{"href":"https://api.github.com/repos/apache/flink/statuses/923fe52cd0998e668e2bc27aeff27e8067eaddab"}}}}