{"action":"created","issue":{"url":"https://api.github.com/repos/caskdata/cdap/issues/3307","labels_url":"https://api.github.com/repos/caskdata/cdap/issues/3307/labels{/name}","comments_url":"https://api.github.com/repos/caskdata/cdap/issues/3307/comments","events_url":"https://api.github.com/repos/caskdata/cdap/issues/3307/events","html_url":"https://github.com/caskdata/cdap/pull/3307","id":97367333,"number":3307,"title":"[CDAP-1983] Added an application for processing and analyzing Wikipedia data using a CDAP Workflow.","user":{"login":"bdmogal","id":3895099,"avatar_url":"https://avatars.githubusercontent.com/u/3895099?v=3","gravatar_id":"","url":"https://api.github.com/users/bdmogal","html_url":"https://github.com/bdmogal","followers_url":"https://api.github.com/users/bdmogal/followers","following_url":"https://api.github.com/users/bdmogal/following{/other_user}","gists_url":"https://api.github.com/users/bdmogal/gists{/gist_id}","starred_url":"https://api.github.com/users/bdmogal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdmogal/subscriptions","organizations_url":"https://api.github.com/users/bdmogal/orgs","repos_url":"https://api.github.com/users/bdmogal/repos","events_url":"https://api.github.com/users/bdmogal/events{/privacy}","received_events_url":"https://api.github.com/users/bdmogal/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/caskdata/cdap/labels/3.1","name":"3.1","color":"14d940"},{"url":"https://api.github.com/repos/caskdata/cdap/labels/Docs","name":"Docs","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-07-27T01:46:02Z","updated_at":"2015-07-27T19:55:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/caskdata/cdap/pulls/3307","html_url":"https://github.com/caskdata/cdap/pull/3307","diff_url":"https://github.com/caskdata/cdap/pull/3307.diff","patch_url":"https://github.com/caskdata/cdap/pull/3307.patch"},"body":"The app contains a CDAP Workflow that runs in either online or offline mode. In the offline mode, it expects Wikipedia data to be available in a Stream. In the online mode, it attempts to download wikipedia data for a provided set of page titles (the output of the Facebook Likes API). Once wikipedia data is available it runs a map-only job to filter/normalize data formatted as text/wiki-text into text/plain. It then runs two analyses on it in a fork -\r\n1. A Spark program that does topic modeling using Latent Dirichlet Allocation (LDA).\r\n2. A MapReduce program that does a simple 'Top N' word generation.\r\n\r\nThis app also demonstrates Workflow Token usage in:\r\n1. Condition Predicates\r\n2. Determining Job configuration (setting it based on values in the token)\r\n3. map/reduce functions\r\n4. Spark Programs\r\n\r\nIt also shows usage of getDataset + put() in Spark programs.\r\n\r\nBuild: http://builds.cask.co/browse/CDAP-RBT355\r\n\r\n\r\nPending/WIP:\r\n- [ ] The app currently does not have a good way to explore output or a service to retrieve results. The LDA output dataset does not have a schema set (it currently has a column per term, so the schema is variable). However, since the aim of the app is to demonstrate workflows, this can be fixed in future.\r\n- [ ] An example of how to use workflow tokens in a Java Spark program.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/3895099/8898257/45699f34-33d6-11e5-8218-374489956fb3.png)\r\n"},"comment":{"url":"https://api.github.com/repos/caskdata/cdap/issues/comments/125325631","html_url":"https://github.com/caskdata/cdap/pull/3307#issuecomment-125325631","issue_url":"https://api.github.com/repos/caskdata/cdap/issues/3307","id":125325631,"user":{"login":"bdmogal","id":3895099,"avatar_url":"https://avatars.githubusercontent.com/u/3895099?v=3","gravatar_id":"","url":"https://api.github.com/users/bdmogal","html_url":"https://github.com/bdmogal","followers_url":"https://api.github.com/users/bdmogal/followers","following_url":"https://api.github.com/users/bdmogal/following{/other_user}","gists_url":"https://api.github.com/users/bdmogal/gists{/gist_id}","starred_url":"https://api.github.com/users/bdmogal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdmogal/subscriptions","organizations_url":"https://api.github.com/users/bdmogal/orgs","repos_url":"https://api.github.com/users/bdmogal/repos","events_url":"https://api.github.com/users/bdmogal/events{/privacy}","received_events_url":"https://api.github.com/users/bdmogal/received_events","type":"User","site_admin":false},"created_at":"2015-07-27T19:55:42Z","updated_at":"2015-07-27T19:55:42Z","body":"@awholegunch fixed documentation in latest commit. Can you please check again?"}}