{"action":"created","issue":{"url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172","labels_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172/labels{/name}","comments_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172/comments","events_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172/events","html_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172","id":57999280,"number":172,"title":"[SPARKR-187] DataFrame to RRDD conversion","user":{"login":"cafreeman","id":7971393,"avatar_url":"https://avatars.githubusercontent.com/u/7971393?v=3","gravatar_id":"","url":"https://api.github.com/users/cafreeman","html_url":"https://github.com/cafreeman","followers_url":"https://api.github.com/users/cafreeman/followers","following_url":"https://api.github.com/users/cafreeman/following{/other_user}","gists_url":"https://api.github.com/users/cafreeman/gists{/gist_id}","starred_url":"https://api.github.com/users/cafreeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cafreeman/subscriptions","organizations_url":"https://api.github.com/users/cafreeman/orgs","repos_url":"https://api.github.com/users/cafreeman/repos","events_url":"https://api.github.com/users/cafreeman/events{/privacy}","received_events_url":"https://api.github.com/users/cafreeman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-02-17T23:25:42Z","updated_at":"2015-02-25T02:00:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172","html_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172","diff_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172.diff","patch_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172.patch"},"body":"This PR extends the initial work done on the Spark 1.3 DataFrame API by  providing an underlying architecture for converting Spark DataFrames to RRDDs as well as to R data.frames. The conversion to RRDD enables the use of UDFs in lapply (and lapply-like) functions on DataFrames. RRDDs converted from Data Frames will also maintain column names, allowing for UDFs that reference specific columns in the DataFrame as well as the ability to create new columns within the UDF using R's built-in $ operator.\r\n\r\nDue to the new serialization format, the \"serialized\" attribute of the RRDD had to be changed from a boolean to a string. This has broad consequences for the rest of the SparkR codebase due to there being a large number of control flow statements that rely on serialized being boolean.  As a result, RDD's class and constructor definitions have been updated and the underlying utils.R, worker.R, and RRDD.scala components have been updated to reflect this change as well.\r\n\r\nIn addition, this PR includes the ability to convert a Spark DataFrame to an R data.frame via the `collectToDF()` function. `collectToDF()` calls its own set of Scala functions and re-orients the data in a DataFrame so that it's organized by column instead of row. This enables R to deserialize the data in column-major order (as opposed to Spark's row-major order) and facilitates much faster loading times."},"comment":{"url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/comments/75891483","html_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172#issuecomment-75891483","issue_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172","id":75891483,"user":{"login":"cafreeman","id":7971393,"avatar_url":"https://avatars.githubusercontent.com/u/7971393?v=3","gravatar_id":"","url":"https://api.github.com/users/cafreeman","html_url":"https://github.com/cafreeman","followers_url":"https://api.github.com/users/cafreeman/followers","following_url":"https://api.github.com/users/cafreeman/following{/other_user}","gists_url":"https://api.github.com/users/cafreeman/gists{/gist_id}","starred_url":"https://api.github.com/users/cafreeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cafreeman/subscriptions","organizations_url":"https://api.github.com/users/cafreeman/orgs","repos_url":"https://api.github.com/users/cafreeman/repos","events_url":"https://api.github.com/users/cafreeman/events{/privacy}","received_events_url":"https://api.github.com/users/cafreeman/received_events","type":"User","site_admin":false},"created_at":"2015-02-25T02:00:33Z","updated_at":"2015-02-25T02:00:33Z","body":"I've implemented all the changes discussed in our conversation earlier today. The `colNames` slot is now a thing of the past and everything has been updated accordingly.\r\n\r\nI also added a `limit` function to the `DataFrame` class and reworked `take` so that it would make use of the catalyst optimizer instead of converting to an RDD and using the RDD version of `take`.\r\n\r\nAt this point, all tests pass and I believe we should be good to go unless anyone else has other issues.\r\n\r\nThanks again for all the feedback @shivaram @concretevitamin @davies "}}