{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/35827501","id":35827501,"diff_hunk":"@@ -1152,9 +1156,385 @@ def sparse(numRows, numCols, colPtrs, rowIndices, values):\n         return SparseMatrix(numRows, numCols, colPtrs, rowIndices, values)\n \n \n+class DistributedMatrix(object):\n+    \"\"\"\n+    Represents a distributively stored matrix backed by one or\n+    more RDDs.\n+\n+    \"\"\"\n+    def numRows(self):\n+        \"\"\"Get or compute the number of rows.\"\"\"\n+        raise NotImplementedError\n+\n+    def numCols(self):\n+        \"\"\"Get or compute the number of cols.\"\"\"\n+        raise NotImplementedError\n+\n+\n+class RowMatrix(DistributedMatrix):\n+    \"\"\"\n+    Represents a row-oriented distributed Matrix with no meaningful\n+    row indices.\n+\n+    .. note:: Experimental\n+    \"\"\"\n+    def __init__(self, rows, numRows=0, numCols=0):\n+        \"\"\"\n+        Create a wrapper over a Java RowMatrix.\n+\n+        :param rows: An RDD of Vectors.\n+        \"\"\"\n+        if not isinstance(rows, RDD):\n+            raise TypeError(\"rows should be an RDD object, got %s\" % type(rows))\n+        first = rows.first()\n+        if not isinstance(first, Vector):\n+            raise TypeError(\"rows should be an RDD of Vectors, got an RDD of %s\" % type(first))\n+\n+        javaRowMatrix = callMLlibFunc(\"createRowMatrix\", rows, long(numRows), int(numCols))\n+        self._jrm = JavaModelWrapper(javaRowMatrix)\n+        self.rows = rows\n+\n+    @staticmethod\n+    def _from_java(javaRowMatrix):","path":"python/pyspark/mllib/linalg.py","position":65,"original_position":61,"commit_id":"4ac646bacc081da6665a7af6a1c6a60de13e590b","original_commit_id":"969a40361c9197368de8c7e1a6bee007e8607f70","user":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"body":"Yeah I agree, it's not ideal, but it does keep the API clean.  An alternative would be to add an optional Java model parameter to each of the class constructors, and use it if present.  Then we could just construct the Java model on the Scala/Java side, grab it's rows, and pass both of those into the class constructors, bypassing the final step of reconstruction.  This parameter wouldn't be very useful for the end user though.\r\n\r\n@mengxr Do you have any thoughts on this?","created_at":"2015-07-30T00:23:43Z","updated_at":"2015-07-30T00:23:43Z","html_url":"https://github.com/apache/spark/pull/7554#discussion_r35827501","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/7554","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/35827501"},"html":{"href":"https://github.com/apache/spark/pull/7554#discussion_r35827501"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/7554"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7554","id":40416926,"html_url":"https://github.com/apache/spark/pull/7554","diff_url":"https://github.com/apache/spark/pull/7554.diff","patch_url":"https://github.com/apache/spark/pull/7554.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/7554","number":7554,"state":"open","locked":false,"title":"[SPARK-6485] [MLlib] [Python] Add CoordinateMatrix/RowMatrix/IndexedRowMatrix to PySpark.","user":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"body":"This PR adds the RowMatrix, IndexedRowMatrix, and CoordinateMatrix distributed matrices to PySpark.  Each distributed matrix class acts as a wrapper around the Scala/Java counterpart by maintaining a reference to the Java object.  New distributed matrices can be created using factory methods added to DistributedMatrices, which creates the Java distributed matrix and then wraps it with the corresponding PySpark class.  This design allows for simple conversion between the various distributed matrices, and lets us re-use the Scala code.  Serialization between Python and Java is implemented using DataFrames as needed for IndexedRowMatrix and CoordinateMatrix for simplicity.  Associated documentation and unit-tests have also been added.  To facilitate code review, this PR implements access to the rows/entries as RDDs, the number of rows & columns, and conversions between the various distributed matrices (not including BlockMatrix), and does not implement the other linear algebra functions of the matrices, although this will be very simple to add now.","created_at":"2015-07-21T02:23:11Z","updated_at":"2015-07-30T00:23:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"38a055821e717d1dab24cfe59c88eec9658c07d9","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/7554/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/7554/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/7554/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/4ac646bacc081da6665a7af6a1c6a60de13e590b","head":{"label":"dusenberrymw:SPARK-6485_Add_CoordinateMatrix_RowMatrix_IndexedMatrix_to_PySpark","ref":"SPARK-6485_Add_CoordinateMatrix_RowMatrix_IndexedMatrix_to_PySpark","sha":"4ac646bacc081da6665a7af6a1c6a60de13e590b","user":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"repo":{"id":35676174,"name":"spark","full_name":"dusenberrymw/spark","owner":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/dusenberrymw/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/dusenberrymw/spark","forks_url":"https://api.github.com/repos/dusenberrymw/spark/forks","keys_url":"https://api.github.com/repos/dusenberrymw/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dusenberrymw/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dusenberrymw/spark/teams","hooks_url":"https://api.github.com/repos/dusenberrymw/spark/hooks","issue_events_url":"https://api.github.com/repos/dusenberrymw/spark/issues/events{/number}","events_url":"https://api.github.com/repos/dusenberrymw/spark/events","assignees_url":"https://api.github.com/repos/dusenberrymw/spark/assignees{/user}","branches_url":"https://api.github.com/repos/dusenberrymw/spark/branches{/branch}","tags_url":"https://api.github.com/repos/dusenberrymw/spark/tags","blobs_url":"https://api.github.com/repos/dusenberrymw/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dusenberrymw/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dusenberrymw/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/dusenberrymw/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dusenberrymw/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/dusenberrymw/spark/languages","stargazers_url":"https://api.github.com/repos/dusenberrymw/spark/stargazers","contributors_url":"https://api.github.com/repos/dusenberrymw/spark/contributors","subscribers_url":"https://api.github.com/repos/dusenberrymw/spark/subscribers","subscription_url":"https://api.github.com/repos/dusenberrymw/spark/subscription","commits_url":"https://api.github.com/repos/dusenberrymw/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/dusenberrymw/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/dusenberrymw/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/dusenberrymw/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/dusenberrymw/spark/contents/{+path}","compare_url":"https://api.github.com/repos/dusenberrymw/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dusenberrymw/spark/merges","archive_url":"https://api.github.com/repos/dusenberrymw/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dusenberrymw/spark/downloads","issues_url":"https://api.github.com/repos/dusenberrymw/spark/issues{/number}","pulls_url":"https://api.github.com/repos/dusenberrymw/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/dusenberrymw/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/dusenberrymw/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dusenberrymw/spark/labels{/name}","releases_url":"https://api.github.com/repos/dusenberrymw/spark/releases{/id}","created_at":"2015-05-15T13:41:57Z","updated_at":"2015-05-15T13:42:19Z","pushed_at":"2015-07-29T23:46:46Z","git_url":"git://github.com/dusenberrymw/spark.git","ssh_url":"git@github.com:dusenberrymw/spark.git","clone_url":"https://github.com/dusenberrymw/spark.git","svn_url":"https://github.com/dusenberrymw/spark","homepage":null,"size":114932,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"936a96cb31a6dd7d8685bce05103e779ca02e763","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-07-29T22:23:05Z","pushed_at":"2015-07-30T00:19:52Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1615732,"stargazers_count":4901,"watchers_count":4901,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":4440,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":351,"forks":4440,"open_issues":351,"watchers":4901,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/7554"},"html":{"href":"https://github.com/apache/spark/pull/7554"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/7554"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/7554/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/7554/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/7554/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/4ac646bacc081da6665a7af6a1c6a60de13e590b"}}}}