{"action":"created","comment":{"url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/comments/25225908","id":25225908,"diff_hunk":"@@ -1,10 +1,74 @@\n package edu.berkeley.cs.amplab.sparkr\n \n import org.apache.spark.SparkContext\n-import org.apache.spark.sql.{SQLContext}\n+import org.apache.spark.rdd.RDD\n+import org.apache.spark.api.java.JavaRDD\n+import org.apache.spark.sql.{SQLContext, DataFrame, Row}\n+import org.apache.spark.sql.types.{StructType}\n+\n+import edu.berkeley.cs.amplab.sparkr.SerDe._\n+\n+import java.io.ByteArrayOutputStream\n+import java.io.DataOutputStream\n \n object SQLUtils {\n   def createSQLContext(sc: SparkContext): SQLContext = {\n     new SQLContext(sc)\n   }\n+\n+  def getColNames(df: DataFrame): Array[String] = {\n+    df.schema.fields.map(_.name)","path":"pkg/src/src/main/scala/edu/berkeley/cs/amplab/sparkr/SQLUtils.scala","position":21,"original_position":21,"commit_id":"481ae37bd5924ae6cc90df233cd729b55225421f","original_commit_id":"9c77b20c7934e2408a48463c2064c115a7f19419","user":{"login":"cafreeman","id":7971393,"avatar_url":"https://avatars.githubusercontent.com/u/7971393?v=3","gravatar_id":"","url":"https://api.github.com/users/cafreeman","html_url":"https://github.com/cafreeman","followers_url":"https://api.github.com/users/cafreeman/followers","following_url":"https://api.github.com/users/cafreeman/following{/other_user}","gists_url":"https://api.github.com/users/cafreeman/gists{/gist_id}","starred_url":"https://api.github.com/users/cafreeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cafreeman/subscriptions","organizations_url":"https://api.github.com/users/cafreeman/orgs","repos_url":"https://api.github.com/users/cafreeman/repos","events_url":"https://api.github.com/users/cafreeman/events{/privacy}","received_events_url":"https://api.github.com/users/cafreeman/received_events","type":"User","site_admin":false},"body":"I just dropped this functions out and called `columns` with `callJMethod` instead. Makes things even simpler.","created_at":"2015-02-24T03:19:03Z","updated_at":"2015-02-24T03:19:03Z","html_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172#discussion_r25225908","pull_request_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172","_links":{"self":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/comments/25225908"},"html":{"href":"https://github.com/amplab-extras/SparkR-pkg/pull/172#discussion_r25225908"},"pull_request":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172"}}},"pull_request":{"url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172","id":29487935,"html_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172","diff_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172.diff","patch_url":"https://github.com/amplab-extras/SparkR-pkg/pull/172.patch","issue_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172","number":172,"state":"open","locked":false,"title":"[SPARKR-187] DataFrame to RRDD conversion","user":{"login":"cafreeman","id":7971393,"avatar_url":"https://avatars.githubusercontent.com/u/7971393?v=3","gravatar_id":"","url":"https://api.github.com/users/cafreeman","html_url":"https://github.com/cafreeman","followers_url":"https://api.github.com/users/cafreeman/followers","following_url":"https://api.github.com/users/cafreeman/following{/other_user}","gists_url":"https://api.github.com/users/cafreeman/gists{/gist_id}","starred_url":"https://api.github.com/users/cafreeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cafreeman/subscriptions","organizations_url":"https://api.github.com/users/cafreeman/orgs","repos_url":"https://api.github.com/users/cafreeman/repos","events_url":"https://api.github.com/users/cafreeman/events{/privacy}","received_events_url":"https://api.github.com/users/cafreeman/received_events","type":"User","site_admin":false},"body":"This PR extends the initial work done on the Spark 1.3 DataFrame API by  providing an underlying architecture for converting Spark DataFrames to RRDDs as well as to R data.frames. The conversion to RRDD enables the use of UDFs in lapply (and lapply-like) functions on DataFrames. RRDDs converted from Data Frames will also maintain column names, allowing for UDFs that reference specific columns in the DataFrame as well as the ability to create new columns within the UDF using R's built-in $ operator.\r\n\r\nDue to the new serialization format, the \"serialized\" attribute of the RRDD had to be changed from a boolean to a string. This has broad consequences for the rest of the SparkR codebase due to there being a large number of control flow statements that rely on serialized being boolean.  As a result, RDD's class and constructor definitions have been updated and the underlying utils.R, worker.R, and RRDD.scala components have been updated to reflect this change as well.\r\n\r\nIn addition, this PR includes the ability to convert a Spark DataFrame to an R data.frame via the `collectToDF()` function. `collectToDF()` calls its own set of Scala functions and re-orients the data in a DataFrame so that it's organized by column instead of row. This enables R to deserialize the data in column-major order (as opposed to Spark's row-major order) and facilitates much faster loading times.","created_at":"2015-02-17T23:25:42Z","updated_at":"2015-02-24T03:19:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c219a1ec6d9b6a3ca57df2cf9a50a1ea26fa6fb7","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172/commits","review_comments_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172/comments","review_comment_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/comments{/number}","comments_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172/comments","statuses_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/statuses/481ae37bd5924ae6cc90df233cd729b55225421f","head":{"label":"cafreeman:sparkr-sql","ref":"sparkr-sql","sha":"481ae37bd5924ae6cc90df233cd729b55225421f","user":{"login":"cafreeman","id":7971393,"avatar_url":"https://avatars.githubusercontent.com/u/7971393?v=3","gravatar_id":"","url":"https://api.github.com/users/cafreeman","html_url":"https://github.com/cafreeman","followers_url":"https://api.github.com/users/cafreeman/followers","following_url":"https://api.github.com/users/cafreeman/following{/other_user}","gists_url":"https://api.github.com/users/cafreeman/gists{/gist_id}","starred_url":"https://api.github.com/users/cafreeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cafreeman/subscriptions","organizations_url":"https://api.github.com/users/cafreeman/orgs","repos_url":"https://api.github.com/users/cafreeman/repos","events_url":"https://api.github.com/users/cafreeman/events{/privacy}","received_events_url":"https://api.github.com/users/cafreeman/received_events","type":"User","site_admin":false},"repo":{"id":27482470,"name":"SparkR-pkg","full_name":"cafreeman/SparkR-pkg","owner":{"login":"cafreeman","id":7971393,"avatar_url":"https://avatars.githubusercontent.com/u/7971393?v=3","gravatar_id":"","url":"https://api.github.com/users/cafreeman","html_url":"https://github.com/cafreeman","followers_url":"https://api.github.com/users/cafreeman/followers","following_url":"https://api.github.com/users/cafreeman/following{/other_user}","gists_url":"https://api.github.com/users/cafreeman/gists{/gist_id}","starred_url":"https://api.github.com/users/cafreeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cafreeman/subscriptions","organizations_url":"https://api.github.com/users/cafreeman/orgs","repos_url":"https://api.github.com/users/cafreeman/repos","events_url":"https://api.github.com/users/cafreeman/events{/privacy}","received_events_url":"https://api.github.com/users/cafreeman/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/cafreeman/SparkR-pkg","description":"R frontend for Spark","fork":true,"url":"https://api.github.com/repos/cafreeman/SparkR-pkg","forks_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/forks","keys_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/teams","hooks_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/hooks","issue_events_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/issues/events{/number}","events_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/events","assignees_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/assignees{/user}","branches_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/branches{/branch}","tags_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/tags","blobs_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/git/refs{/sha}","trees_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/statuses/{sha}","languages_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/languages","stargazers_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/stargazers","contributors_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/contributors","subscribers_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/subscribers","subscription_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/subscription","commits_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/commits{/sha}","git_commits_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/git/commits{/sha}","comments_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/comments{/number}","issue_comment_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/issues/comments{/number}","contents_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/contents/{+path}","compare_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/merges","archive_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/downloads","issues_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/issues{/number}","pulls_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/pulls{/number}","milestones_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/milestones{/number}","notifications_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/labels{/name}","releases_url":"https://api.github.com/repos/cafreeman/SparkR-pkg/releases{/id}","created_at":"2014-12-03T10:49:30Z","updated_at":"2015-02-16T16:33:05Z","pushed_at":"2015-02-23T15:56:40Z","git_url":"git://github.com/cafreeman/SparkR-pkg.git","ssh_url":"git@github.com:cafreeman/SparkR-pkg.git","clone_url":"https://github.com/cafreeman/SparkR-pkg.git","svn_url":"https://github.com/cafreeman/SparkR-pkg","homepage":"http://amplab-extras.github.io/SparkR-pkg","size":3651,"stargazers_count":0,"watchers_count":0,"language":"R","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"amplab-extras:sparkr-sql","ref":"sparkr-sql","sha":"e2a75603a89778b95214342736cb2b8e71f8b8b0","user":{"login":"amplab-extras","id":6403540,"avatar_url":"https://avatars.githubusercontent.com/u/6403540?v=3","gravatar_id":"","url":"https://api.github.com/users/amplab-extras","html_url":"https://github.com/amplab-extras","followers_url":"https://api.github.com/users/amplab-extras/followers","following_url":"https://api.github.com/users/amplab-extras/following{/other_user}","gists_url":"https://api.github.com/users/amplab-extras/gists{/gist_id}","starred_url":"https://api.github.com/users/amplab-extras/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amplab-extras/subscriptions","organizations_url":"https://api.github.com/users/amplab-extras/orgs","repos_url":"https://api.github.com/users/amplab-extras/repos","events_url":"https://api.github.com/users/amplab-extras/events{/privacy}","received_events_url":"https://api.github.com/users/amplab-extras/received_events","type":"Organization","site_admin":false},"repo":{"id":15365789,"name":"SparkR-pkg","full_name":"amplab-extras/SparkR-pkg","owner":{"login":"amplab-extras","id":6403540,"avatar_url":"https://avatars.githubusercontent.com/u/6403540?v=3","gravatar_id":"","url":"https://api.github.com/users/amplab-extras","html_url":"https://github.com/amplab-extras","followers_url":"https://api.github.com/users/amplab-extras/followers","following_url":"https://api.github.com/users/amplab-extras/following{/other_user}","gists_url":"https://api.github.com/users/amplab-extras/gists{/gist_id}","starred_url":"https://api.github.com/users/amplab-extras/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amplab-extras/subscriptions","organizations_url":"https://api.github.com/users/amplab-extras/orgs","repos_url":"https://api.github.com/users/amplab-extras/repos","events_url":"https://api.github.com/users/amplab-extras/events{/privacy}","received_events_url":"https://api.github.com/users/amplab-extras/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/amplab-extras/SparkR-pkg","description":"R frontend for Spark","fork":false,"url":"https://api.github.com/repos/amplab-extras/SparkR-pkg","forks_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/forks","keys_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/teams","hooks_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/hooks","issue_events_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/events{/number}","events_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/events","assignees_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/assignees{/user}","branches_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/branches{/branch}","tags_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/tags","blobs_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/git/refs{/sha}","trees_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/statuses/{sha}","languages_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/languages","stargazers_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/stargazers","contributors_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/contributors","subscribers_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/subscribers","subscription_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/subscription","commits_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/commits{/sha}","git_commits_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/git/commits{/sha}","comments_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/comments{/number}","issue_comment_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/comments{/number}","contents_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/contents/{+path}","compare_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/merges","archive_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/downloads","issues_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues{/number}","pulls_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls{/number}","milestones_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/milestones{/number}","notifications_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/labels{/name}","releases_url":"https://api.github.com/repos/amplab-extras/SparkR-pkg/releases{/id}","created_at":"2013-12-21T21:16:54Z","updated_at":"2015-02-21T10:47:17Z","pushed_at":"2015-02-21T01:55:48Z","git_url":"git://github.com/amplab-extras/SparkR-pkg.git","ssh_url":"git@github.com:amplab-extras/SparkR-pkg.git","clone_url":"https://github.com/amplab-extras/SparkR-pkg.git","svn_url":"https://github.com/amplab-extras/SparkR-pkg","homepage":"http://amplab-extras.github.io/SparkR-pkg","size":5360,"stargazers_count":399,"watchers_count":399,"language":"R","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":139,"mirror_url":null,"open_issues_count":22,"forks":139,"open_issues":22,"watchers":399,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172"},"html":{"href":"https://github.com/amplab-extras/SparkR-pkg/pull/172"},"issue":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172"},"comments":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/issues/172/comments"},"review_comments":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172/comments"},"review_comment":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/pulls/172/commits"},"statuses":{"href":"https://api.github.com/repos/amplab-extras/SparkR-pkg/statuses/481ae37bd5924ae6cc90df233cd729b55225421f"}}}}