{"action":"created","issue":{"url":"https://api.github.com/repos/hoxo-m/SparkRext/issues/1","labels_url":"https://api.github.com/repos/hoxo-m/SparkRext/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/hoxo-m/SparkRext/issues/1/comments","events_url":"https://api.github.com/repos/hoxo-m/SparkRext/issues/1/events","html_url":"https://github.com/hoxo-m/SparkRext/pull/1","id":112888797,"number":1,"title":"Override dplyr generics instead","user":{"login":"saurfang","id":4317392,"avatar_url":"https://avatars.githubusercontent.com/u/4317392?v=3","gravatar_id":"","url":"https://api.github.com/users/saurfang","html_url":"https://github.com/saurfang","followers_url":"https://api.github.com/users/saurfang/followers","following_url":"https://api.github.com/users/saurfang/following{/other_user}","gists_url":"https://api.github.com/users/saurfang/gists{/gist_id}","starred_url":"https://api.github.com/users/saurfang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saurfang/subscriptions","organizations_url":"https://api.github.com/users/saurfang/orgs","repos_url":"https://api.github.com/users/saurfang/repos","events_url":"https://api.github.com/users/saurfang/events{/privacy}","received_events_url":"https://api.github.com/users/saurfang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-22T20:50:57Z","updated_at":"2015-10-23T17:35:35Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/hoxo-m/SparkRext/pulls/1","html_url":"https://github.com/hoxo-m/SparkRext/pull/1","diff_url":"https://github.com/hoxo-m/SparkRext/pull/1.diff","patch_url":"https://github.com/hoxo-m/SparkRext/pull/1.patch"},"body":"Hi Koji,\r\n\r\nThanks a lot for this package and I couldn't agree more on the expressiveness and succinctness dplyr syntax can bring to SparkR.\r\n\r\nI realize you have created a separate branch called `nodplyr` where you try to drop dependency on dplyr. I have a slightly different view that it'd be better embrace dplyr instead. I find it to be a shame if we restrict ourselves to SparkR alone for any data wrangling. Rather I believe that to conduct effective data science project, you need to do well in both spectrum: both big data and small data. This is especially true when you are visualizing dataset in different way. You can see some examples I included here:\r\nhttps://github.com/saurfang/SparkRext/blob/dplyr/README.md#5-2-interoperability-betwen-sparkr-and-dplyr\r\n\r\nTo achieve this goal, I imported all top-level verbs from dplyr but overrode Standard Evaluation generics instead (e.g. `select_`, `filter_`). So that you preserve all dplyr functionality while function calls work exactly the same for SparkR DataFrame.\r\n\r\nI also made some minor changes around typos and Spark 1.5's new feature where it is able to handle timestamp correctly. Finally I dropped the import of pipe since I feel that while pipeR is much faster but the speed gain is negligible when you are operating a heavy machinery like SparkR where each job takes more than seconds.\r\n\r\n\r\nAs I said above, I recognize you might have different thoughts on whether there is value to be more dplyr / small data friendly. I'm curious to hear what you think about this idea. Thanks again for the awesome idea and package!\r\n\r\nForest"},"comment":{"url":"https://api.github.com/repos/hoxo-m/SparkRext/issues/comments/150641464","html_url":"https://github.com/hoxo-m/SparkRext/pull/1#issuecomment-150641464","issue_url":"https://api.github.com/repos/hoxo-m/SparkRext/issues/1","id":150641464,"user":{"login":"saurfang","id":4317392,"avatar_url":"https://avatars.githubusercontent.com/u/4317392?v=3","gravatar_id":"","url":"https://api.github.com/users/saurfang","html_url":"https://github.com/saurfang","followers_url":"https://api.github.com/users/saurfang/followers","following_url":"https://api.github.com/users/saurfang/following{/other_user}","gists_url":"https://api.github.com/users/saurfang/gists{/gist_id}","starred_url":"https://api.github.com/users/saurfang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saurfang/subscriptions","organizations_url":"https://api.github.com/users/saurfang/orgs","repos_url":"https://api.github.com/users/saurfang/repos","events_url":"https://api.github.com/users/saurfang/events{/privacy}","received_events_url":"https://api.github.com/users/saurfang/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T17:35:35Z","updated_at":"2015-10-23T17:35:35Z","body":"This is awesome. Thanks for sharing! Direct extension to dplyr definitely makes the package more likely to be stable but take advantage of new features as dplyr continue evolving."}}