{"action":"created","comment":{"url":"https://api.github.com/repos/caskdata/cdap/pulls/comments/35581237","id":35581237,"diff_hunk":"@@ -0,0 +1,140 @@\n+.. meta::\r\n+    :author: Cask Data, Inc.\r\n+    :description: Cask Data Application Platform Wikipedia Pipeline Application\r\n+    :copyright: Copyright © 2015 Cask Data, Inc.\r\n+\r\n+.. _examples-wikipedia-data-pipeline:\r\n+\r\n+==================\r\n+Wikipedia Pipeline\r\n+==================\r\n+\r\n+A Cask Data Application Platform (CDAP) example demonstrating a typical data processing pipeline using CDAP Workflows.\r\n+\r\n+Overview\r\n+========\r\n+\r\n+This example demonstrates a CDAP application performing analysis on Wikipedia data using MapReduce and Spark programs\r\n+running within a CDAP Workflow: *WikipediaPipelineWorkflow*.\r\n+\r\n+This example can be run in both online and offline modes.\r\n+- In the online mode, the custom action *DownloadWikiDataAction* reads the *pageTitleStream* is one output element\r\n+of the `Facebook \"likes\" API <https://developers.facebook.com/docs/graph-api/reference/v2.4/object/likes>`__.\r\n+For each event, it tries to download Wikipedia data for the page using the\r\n+`MediaWiki Wikipedia API <https://www.mediawiki.org/wiki/API:Main_page>`__. It stores the downloaded data in the\r\n+KeyValueTable dataset *wikiData*.\r\n+- In the offline mode, it expects Wikipedia data formatted per the output of the MediaWiki API in the stream *wikiStream*.\r\n+\r\n+The MapReduce program *wikiDataToDataset* consumes this stream and stores it in the same KeyValueTable dataset\r\n+ *wikiData*. Data can be uploaded to the *wikiStream* using the CDAP CLI.\r\n+\r\n+Once raw Wikipedia data is available is available from using either the online or offline modes, the\r\n+*WikipediaPipelineWorkflow* runs a MapReduce program *WikiContentValidatorAndNormalizer* that filters bad records from\r\n+the raw data, as well as normalizes it by converting the wikitext-formatted data to plain text. It then stores the\r\n+output in another KeyValueTable dataset *normalized*.\r\n+\r\n+The *WikipediaPipelineWorkflow* then contains a fork, with two branches. One branch runs the Apache Spark program *SparkWikipediaAnalyzer*.\r\n+This program consumes normalized data and runs topic modeling on it using the Latent Dirichlet Allocation (LDA) algorithm. It stores its\r\n+output in the CDAP Table dataset *lda*, with one row for each iteration, and a column per topic containing the score for that topic.\r\n+The other branch contains a MapReduce program *TopNMapReduce* that consumes the normalized data and produces the top N words in the dataset *topn*.\r\n+\r\n+Let's look at some of these components, and then run the application and see the results.\r\n+\r\n+The WikipediaPipeline Application\r\n+---------------------------------\r\n+\r\n+As in the other `examples <index.html>`__, the components\r\n+of the application are tied together by the class ``WikipediaPipelineApp``:\r\n+\r\n+.. literalinclude:: /../../../cdap-examples/WikipediaPipeline/src/main/java/co/cask/cdap/examples/wikipedia/WikipediaPipelineApp.java\r\n+   :language: java\r\n+   :lines: 24-55\r\n+\r\n+\r\n+Running the Example\r\n+===================\r\n+\r\n+.. highlight:: console\r\n+\r\n+Injecting data\r\n+--------------\r\n+The *pageTitleStream* consumes events in the format returned by the Facebook \"Likes\" Graph API.\r\n+\r\n+Once the application is deployed:\r\n+\r\n+- Inject a file of \"likes\" data to the stream *pageTitleStream* by running this command from the Standalone CDAP SDK\r\n+directory, using the CDAP Command Line Interface::\r\n+\r\n+  $ cdap-cli.sh load stream pageTitleStream examples/WikipediaPipeline/data/fb-likes.txt\r\n+  Successfully sent stream event to stream 'pageTitleStream'\r\n+\r\n+\r\n+The *wikiStream* consumes events in the format returned by the MediaWiki Wikipedia API.\r\n+\r\n+Once the application is deployed:\r","path":"cdap-docs/examples-manual/source/examples/wikipedia-data-pipeline.rst","position":74,"original_position":74,"commit_id":"c825e376c753e6a732e495e108fa3b6256b1ba26","original_commit_id":"c825e376c753e6a732e495e108fa3b6256b1ba26","user":{"login":"bdmogal","id":3895099,"avatar_url":"https://avatars.githubusercontent.com/u/3895099?v=3","gravatar_id":"","url":"https://api.github.com/users/bdmogal","html_url":"https://github.com/bdmogal","followers_url":"https://api.github.com/users/bdmogal/followers","following_url":"https://api.github.com/users/bdmogal/following{/other_user}","gists_url":"https://api.github.com/users/bdmogal/gists{/gist_id}","starred_url":"https://api.github.com/users/bdmogal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdmogal/subscriptions","organizations_url":"https://api.github.com/users/bdmogal/orgs","repos_url":"https://api.github.com/users/bdmogal/repos","events_url":"https://api.github.com/users/bdmogal/events{/privacy}","received_events_url":"https://api.github.com/users/bdmogal/received_events","type":"User","site_admin":false},"body":"This is there because it is for a different stream. Anyway, I think I'll remove both these lines. I think we can safely assume that the application has been deployed before users run these commands.","created_at":"2015-07-27T20:18:04Z","updated_at":"2015-07-27T20:18:04Z","html_url":"https://github.com/caskdata/cdap/pull/3307#discussion_r35581237","pull_request_url":"https://api.github.com/repos/caskdata/cdap/pulls/3307","_links":{"self":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/comments/35581237"},"html":{"href":"https://github.com/caskdata/cdap/pull/3307#discussion_r35581237"},"pull_request":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/3307"}}},"pull_request":{"url":"https://api.github.com/repos/caskdata/cdap/pulls/3307","id":40864535,"html_url":"https://github.com/caskdata/cdap/pull/3307","diff_url":"https://github.com/caskdata/cdap/pull/3307.diff","patch_url":"https://github.com/caskdata/cdap/pull/3307.patch","issue_url":"https://api.github.com/repos/caskdata/cdap/issues/3307","number":3307,"state":"open","locked":false,"title":"[CDAP-1983] Added an application for processing and analyzing Wikipedia data using a CDAP Workflow.","user":{"login":"bdmogal","id":3895099,"avatar_url":"https://avatars.githubusercontent.com/u/3895099?v=3","gravatar_id":"","url":"https://api.github.com/users/bdmogal","html_url":"https://github.com/bdmogal","followers_url":"https://api.github.com/users/bdmogal/followers","following_url":"https://api.github.com/users/bdmogal/following{/other_user}","gists_url":"https://api.github.com/users/bdmogal/gists{/gist_id}","starred_url":"https://api.github.com/users/bdmogal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdmogal/subscriptions","organizations_url":"https://api.github.com/users/bdmogal/orgs","repos_url":"https://api.github.com/users/bdmogal/repos","events_url":"https://api.github.com/users/bdmogal/events{/privacy}","received_events_url":"https://api.github.com/users/bdmogal/received_events","type":"User","site_admin":false},"body":"The app contains a CDAP Workflow that runs in either online or offline mode. In the offline mode, it expects Wikipedia data to be available in a Stream. In the online mode, it attempts to download wikipedia data for a provided set of page titles (the output of the Facebook Likes API). Once wikipedia data is available it runs a map-only job to filter/normalize data formatted as text/wiki-text into text/plain. It then runs two analyses on it in a fork -\r\n1. A Spark program that does topic modeling using Latent Dirichlet Allocation (LDA).\r\n2. A MapReduce program that does a simple 'Top N' word generation.\r\n\r\nThis app also demonstrates Workflow Token usage in:\r\n1. Condition Predicates\r\n2. Determining Job configuration (setting it based on values in the token)\r\n3. map/reduce functions\r\n4. Spark Programs\r\n\r\nIt also shows usage of getDataset + put() in Spark programs.\r\n\r\nBuild: http://builds.cask.co/browse/CDAP-RBT355\r\n\r\n\r\nPending/WIP:\r\n- [ ] The app currently does not have a good way to explore output or a service to retrieve results. The LDA output dataset does not have a schema set (it currently has a column per term, so the schema is variable). However, since the aim of the app is to demonstrate workflows, this can be fixed in future.\r\n- [ ] An example of how to use workflow tokens in a Java Spark program.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/3895099/8898257/45699f34-33d6-11e5-8218-374489956fb3.png)\r\n","created_at":"2015-07-27T01:46:02Z","updated_at":"2015-07-27T20:18:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"52dedffadd90dbb34d0ebd958a386e6678c0f7ce","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/caskdata/cdap/pulls/3307/commits","review_comments_url":"https://api.github.com/repos/caskdata/cdap/pulls/3307/comments","review_comment_url":"https://api.github.com/repos/caskdata/cdap/pulls/comments{/number}","comments_url":"https://api.github.com/repos/caskdata/cdap/issues/3307/comments","statuses_url":"https://api.github.com/repos/caskdata/cdap/statuses/c825e376c753e6a732e495e108fa3b6256b1ba26","head":{"label":"caskdata:feature/workflow-example","ref":"feature/workflow-example","sha":"c825e376c753e6a732e495e108fa3b6256b1ba26","user":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"repo":{"id":22542759,"name":"cdap","full_name":"caskdata/cdap","owner":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/caskdata/cdap","description":"Cask Data Application Platform (CDAP)","fork":false,"url":"https://api.github.com/repos/caskdata/cdap","forks_url":"https://api.github.com/repos/caskdata/cdap/forks","keys_url":"https://api.github.com/repos/caskdata/cdap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/caskdata/cdap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/caskdata/cdap/teams","hooks_url":"https://api.github.com/repos/caskdata/cdap/hooks","issue_events_url":"https://api.github.com/repos/caskdata/cdap/issues/events{/number}","events_url":"https://api.github.com/repos/caskdata/cdap/events","assignees_url":"https://api.github.com/repos/caskdata/cdap/assignees{/user}","branches_url":"https://api.github.com/repos/caskdata/cdap/branches{/branch}","tags_url":"https://api.github.com/repos/caskdata/cdap/tags","blobs_url":"https://api.github.com/repos/caskdata/cdap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/caskdata/cdap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/caskdata/cdap/git/refs{/sha}","trees_url":"https://api.github.com/repos/caskdata/cdap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/caskdata/cdap/statuses/{sha}","languages_url":"https://api.github.com/repos/caskdata/cdap/languages","stargazers_url":"https://api.github.com/repos/caskdata/cdap/stargazers","contributors_url":"https://api.github.com/repos/caskdata/cdap/contributors","subscribers_url":"https://api.github.com/repos/caskdata/cdap/subscribers","subscription_url":"https://api.github.com/repos/caskdata/cdap/subscription","commits_url":"https://api.github.com/repos/caskdata/cdap/commits{/sha}","git_commits_url":"https://api.github.com/repos/caskdata/cdap/git/commits{/sha}","comments_url":"https://api.github.com/repos/caskdata/cdap/comments{/number}","issue_comment_url":"https://api.github.com/repos/caskdata/cdap/issues/comments{/number}","contents_url":"https://api.github.com/repos/caskdata/cdap/contents/{+path}","compare_url":"https://api.github.com/repos/caskdata/cdap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/caskdata/cdap/merges","archive_url":"https://api.github.com/repos/caskdata/cdap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/caskdata/cdap/downloads","issues_url":"https://api.github.com/repos/caskdata/cdap/issues{/number}","pulls_url":"https://api.github.com/repos/caskdata/cdap/pulls{/number}","milestones_url":"https://api.github.com/repos/caskdata/cdap/milestones{/number}","notifications_url":"https://api.github.com/repos/caskdata/cdap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/caskdata/cdap/labels{/name}","releases_url":"https://api.github.com/repos/caskdata/cdap/releases{/id}","created_at":"2014-08-02T09:00:10Z","updated_at":"2015-06-30T16:06:18Z","pushed_at":"2015-07-27T20:09:42Z","git_url":"git://github.com/caskdata/cdap.git","ssh_url":"git@github.com:caskdata/cdap.git","clone_url":"https://github.com/caskdata/cdap.git","svn_url":"https://github.com/caskdata/cdap","homepage":"http://cdap.io","size":838169,"stargazers_count":56,"watchers_count":56,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":19,"mirror_url":null,"open_issues_count":16,"forks":19,"open_issues":16,"watchers":56,"default_branch":"develop"}},"base":{"label":"caskdata:release/3.1","ref":"release/3.1","sha":"8e3e02fefed42c94a54a6fcd348aeca26cb5893c","user":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"repo":{"id":22542759,"name":"cdap","full_name":"caskdata/cdap","owner":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/caskdata/cdap","description":"Cask Data Application Platform (CDAP)","fork":false,"url":"https://api.github.com/repos/caskdata/cdap","forks_url":"https://api.github.com/repos/caskdata/cdap/forks","keys_url":"https://api.github.com/repos/caskdata/cdap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/caskdata/cdap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/caskdata/cdap/teams","hooks_url":"https://api.github.com/repos/caskdata/cdap/hooks","issue_events_url":"https://api.github.com/repos/caskdata/cdap/issues/events{/number}","events_url":"https://api.github.com/repos/caskdata/cdap/events","assignees_url":"https://api.github.com/repos/caskdata/cdap/assignees{/user}","branches_url":"https://api.github.com/repos/caskdata/cdap/branches{/branch}","tags_url":"https://api.github.com/repos/caskdata/cdap/tags","blobs_url":"https://api.github.com/repos/caskdata/cdap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/caskdata/cdap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/caskdata/cdap/git/refs{/sha}","trees_url":"https://api.github.com/repos/caskdata/cdap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/caskdata/cdap/statuses/{sha}","languages_url":"https://api.github.com/repos/caskdata/cdap/languages","stargazers_url":"https://api.github.com/repos/caskdata/cdap/stargazers","contributors_url":"https://api.github.com/repos/caskdata/cdap/contributors","subscribers_url":"https://api.github.com/repos/caskdata/cdap/subscribers","subscription_url":"https://api.github.com/repos/caskdata/cdap/subscription","commits_url":"https://api.github.com/repos/caskdata/cdap/commits{/sha}","git_commits_url":"https://api.github.com/repos/caskdata/cdap/git/commits{/sha}","comments_url":"https://api.github.com/repos/caskdata/cdap/comments{/number}","issue_comment_url":"https://api.github.com/repos/caskdata/cdap/issues/comments{/number}","contents_url":"https://api.github.com/repos/caskdata/cdap/contents/{+path}","compare_url":"https://api.github.com/repos/caskdata/cdap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/caskdata/cdap/merges","archive_url":"https://api.github.com/repos/caskdata/cdap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/caskdata/cdap/downloads","issues_url":"https://api.github.com/repos/caskdata/cdap/issues{/number}","pulls_url":"https://api.github.com/repos/caskdata/cdap/pulls{/number}","milestones_url":"https://api.github.com/repos/caskdata/cdap/milestones{/number}","notifications_url":"https://api.github.com/repos/caskdata/cdap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/caskdata/cdap/labels{/name}","releases_url":"https://api.github.com/repos/caskdata/cdap/releases{/id}","created_at":"2014-08-02T09:00:10Z","updated_at":"2015-06-30T16:06:18Z","pushed_at":"2015-07-27T20:09:42Z","git_url":"git://github.com/caskdata/cdap.git","ssh_url":"git@github.com:caskdata/cdap.git","clone_url":"https://github.com/caskdata/cdap.git","svn_url":"https://github.com/caskdata/cdap","homepage":"http://cdap.io","size":838169,"stargazers_count":56,"watchers_count":56,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":19,"mirror_url":null,"open_issues_count":16,"forks":19,"open_issues":16,"watchers":56,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/3307"},"html":{"href":"https://github.com/caskdata/cdap/pull/3307"},"issue":{"href":"https://api.github.com/repos/caskdata/cdap/issues/3307"},"comments":{"href":"https://api.github.com/repos/caskdata/cdap/issues/3307/comments"},"review_comments":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/3307/comments"},"review_comment":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/3307/commits"},"statuses":{"href":"https://api.github.com/repos/caskdata/cdap/statuses/c825e376c753e6a732e495e108fa3b6256b1ba26"}}}}