{"action":"created","comment":{"url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/comments/39343803","id":39343803,"diff_hunk":"@@ -0,0 +1,49 @@\n+# Proposal for Incubation\n+\n+## Subproject name\n+\n+sparkmagic\n+\n+## Development team and Advocate\n+\n+Subproject development team:\n+\n+* Alejandro Guerrero Gonzalez, Microsoft (`@aggFTW`)\n+* Joel Zambrano, Microsoft (`@djoelz`)\n+* Auberon Lopez, Cal Poly (`@alope107`)\n+\n+Steering Council Advocate\n+\n+* Brian Granger (`@ellisonbg`)\n+\n+## Subproject goals, scope and functionality\n+\n+Goals:\n+* Make Spark usage from the IPython notebook as frictionless as possible by decoupling Spark code execution from Spark installation.\n+* These magics will be packaged so that they can be pip-installable or conda-installable.\n+* Allow any Jupyter installation to point to different Spark installations.\n+\n+Scope:\n+* Spark will be the first back end provided by these magics, but it could easily be extended to other big data back ends.\n+* Remote Spark execution through a REST endpoint which allows for Scala, Python, and R support as of Sept 2015.\n+\n+Functionality:\n+* Enable users to remotely connect to a REST Spark back end.\n+* Return of pandas dataframes from Spark when appropriate. This will allow users to integrate spark with the Python libraries available for pandas dataframes.\n+\n+## Audience\n+\n+Spark users who want to have the best experience possible from a Notebook. These users care about the libraries ecosystem the Jupyter community has created.\n+\n+## Other options\n+\n+Alternatives we know of:\n+\n+* Combination of IPython, R kernel (or rpy2), and Scala kernel for an in-Spark-cluster Jupyter installation. This does not allow the user to point to different Spark clusters. It might also result in resource contention (CPU or memory) between the Jupyter installation and Spark.\n+* IBM's Spark kernel does not provide a REST endpoint, requires the installation of Jupyter in the cluster, and does not create pandas dataframes.\n+* sparklingpandas builds on Spark's DataFrame class to give users a Pandas-like API but does not take care of remote connectivity and is for pyspark only.\n+\n+## Integration with Project Jupyter\n+\n+These magics will be pip-installable and loadable from any Jupyter/IPython installation.\n+By virtue of returning pandas dataframes, the dataframes will be easily visualizable by using the library created by the automatic rich visualizations incubation subproject.","path":"sparkmagic/proposal.md","position":49,"original_position":49,"commit_id":"55e445f6468c8fce714daebc58333b648e5b25f2","original_commit_id":"55e445f6468c8fce714daebc58333b648e5b25f2","user":{"login":"aggFTW","id":654204,"avatar_url":"https://avatars.githubusercontent.com/u/654204?v=3","gravatar_id":"","url":"https://api.github.com/users/aggFTW","html_url":"https://github.com/aggFTW","followers_url":"https://api.github.com/users/aggFTW/followers","following_url":"https://api.github.com/users/aggFTW/following{/other_user}","gists_url":"https://api.github.com/users/aggFTW/gists{/gist_id}","starred_url":"https://api.github.com/users/aggFTW/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aggFTW/subscriptions","organizations_url":"https://api.github.com/users/aggFTW/orgs","repos_url":"https://api.github.com/users/aggFTW/repos","events_url":"https://api.github.com/users/aggFTW/events{/privacy}","received_events_url":"https://api.github.com/users/aggFTW/received_events","type":"User","site_admin":false},"body":"Thanks @damianavila! I'll add the links.","created_at":"2015-09-13T02:33:03Z","updated_at":"2015-09-13T02:33:03Z","html_url":"https://github.com/jupyter-incubator/proposals/pull/2#discussion_r39343803","pull_request_url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2","_links":{"self":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/comments/39343803"},"html":{"href":"https://github.com/jupyter-incubator/proposals/pull/2#discussion_r39343803"},"pull_request":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2"}}},"pull_request":{"url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2","id":44839585,"html_url":"https://github.com/jupyter-incubator/proposals/pull/2","diff_url":"https://github.com/jupyter-incubator/proposals/pull/2.diff","patch_url":"https://github.com/jupyter-incubator/proposals/pull/2.patch","issue_url":"https://api.github.com/repos/jupyter-incubator/proposals/issues/2","number":2,"state":"open","locked":false,"title":"sparkmagic proposal","user":{"login":"aggFTW","id":654204,"avatar_url":"https://avatars.githubusercontent.com/u/654204?v=3","gravatar_id":"","url":"https://api.github.com/users/aggFTW","html_url":"https://github.com/aggFTW","followers_url":"https://api.github.com/users/aggFTW/followers","following_url":"https://api.github.com/users/aggFTW/following{/other_user}","gists_url":"https://api.github.com/users/aggFTW/gists{/gist_id}","starred_url":"https://api.github.com/users/aggFTW/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aggFTW/subscriptions","organizations_url":"https://api.github.com/users/aggFTW/orgs","repos_url":"https://api.github.com/users/aggFTW/repos","events_url":"https://api.github.com/users/aggFTW/events{/privacy}","received_events_url":"https://api.github.com/users/aggFTW/received_events","type":"User","site_admin":false},"body":"sparkmagic incubation proposal.\r\n\r\nLink to repo in proposals.md to be added once repo is created.\r\n\r\n","created_at":"2015-09-11T23:29:00Z","updated_at":"2015-09-13T02:33:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c8c9a83108060f6b2b0c5ef4fff81be32280daf9","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2/commits","review_comments_url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2/comments","review_comment_url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/comments{/number}","comments_url":"https://api.github.com/repos/jupyter-incubator/proposals/issues/2/comments","statuses_url":"https://api.github.com/repos/jupyter-incubator/proposals/statuses/55e445f6468c8fce714daebc58333b648e5b25f2","head":{"label":"aggFTW:master","ref":"master","sha":"55e445f6468c8fce714daebc58333b648e5b25f2","user":{"login":"aggFTW","id":654204,"avatar_url":"https://avatars.githubusercontent.com/u/654204?v=3","gravatar_id":"","url":"https://api.github.com/users/aggFTW","html_url":"https://github.com/aggFTW","followers_url":"https://api.github.com/users/aggFTW/followers","following_url":"https://api.github.com/users/aggFTW/following{/other_user}","gists_url":"https://api.github.com/users/aggFTW/gists{/gist_id}","starred_url":"https://api.github.com/users/aggFTW/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aggFTW/subscriptions","organizations_url":"https://api.github.com/users/aggFTW/orgs","repos_url":"https://api.github.com/users/aggFTW/repos","events_url":"https://api.github.com/users/aggFTW/events{/privacy}","received_events_url":"https://api.github.com/users/aggFTW/received_events","type":"User","site_admin":false},"repo":{"id":42336735,"name":"proposals","full_name":"aggFTW/proposals","owner":{"login":"aggFTW","id":654204,"avatar_url":"https://avatars.githubusercontent.com/u/654204?v=3","gravatar_id":"","url":"https://api.github.com/users/aggFTW","html_url":"https://github.com/aggFTW","followers_url":"https://api.github.com/users/aggFTW/followers","following_url":"https://api.github.com/users/aggFTW/following{/other_user}","gists_url":"https://api.github.com/users/aggFTW/gists{/gist_id}","starred_url":"https://api.github.com/users/aggFTW/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aggFTW/subscriptions","organizations_url":"https://api.github.com/users/aggFTW/orgs","repos_url":"https://api.github.com/users/aggFTW/repos","events_url":"https://api.github.com/users/aggFTW/events{/privacy}","received_events_url":"https://api.github.com/users/aggFTW/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/aggFTW/proposals","description":"Proposals for new Jupyter subprojects to enter into incubation","fork":true,"url":"https://api.github.com/repos/aggFTW/proposals","forks_url":"https://api.github.com/repos/aggFTW/proposals/forks","keys_url":"https://api.github.com/repos/aggFTW/proposals/keys{/key_id}","collaborators_url":"https://api.github.com/repos/aggFTW/proposals/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/aggFTW/proposals/teams","hooks_url":"https://api.github.com/repos/aggFTW/proposals/hooks","issue_events_url":"https://api.github.com/repos/aggFTW/proposals/issues/events{/number}","events_url":"https://api.github.com/repos/aggFTW/proposals/events","assignees_url":"https://api.github.com/repos/aggFTW/proposals/assignees{/user}","branches_url":"https://api.github.com/repos/aggFTW/proposals/branches{/branch}","tags_url":"https://api.github.com/repos/aggFTW/proposals/tags","blobs_url":"https://api.github.com/repos/aggFTW/proposals/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/aggFTW/proposals/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/aggFTW/proposals/git/refs{/sha}","trees_url":"https://api.github.com/repos/aggFTW/proposals/git/trees{/sha}","statuses_url":"https://api.github.com/repos/aggFTW/proposals/statuses/{sha}","languages_url":"https://api.github.com/repos/aggFTW/proposals/languages","stargazers_url":"https://api.github.com/repos/aggFTW/proposals/stargazers","contributors_url":"https://api.github.com/repos/aggFTW/proposals/contributors","subscribers_url":"https://api.github.com/repos/aggFTW/proposals/subscribers","subscription_url":"https://api.github.com/repos/aggFTW/proposals/subscription","commits_url":"https://api.github.com/repos/aggFTW/proposals/commits{/sha}","git_commits_url":"https://api.github.com/repos/aggFTW/proposals/git/commits{/sha}","comments_url":"https://api.github.com/repos/aggFTW/proposals/comments{/number}","issue_comment_url":"https://api.github.com/repos/aggFTW/proposals/issues/comments{/number}","contents_url":"https://api.github.com/repos/aggFTW/proposals/contents/{+path}","compare_url":"https://api.github.com/repos/aggFTW/proposals/compare/{base}...{head}","merges_url":"https://api.github.com/repos/aggFTW/proposals/merges","archive_url":"https://api.github.com/repos/aggFTW/proposals/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/aggFTW/proposals/downloads","issues_url":"https://api.github.com/repos/aggFTW/proposals/issues{/number}","pulls_url":"https://api.github.com/repos/aggFTW/proposals/pulls{/number}","milestones_url":"https://api.github.com/repos/aggFTW/proposals/milestones{/number}","notifications_url":"https://api.github.com/repos/aggFTW/proposals/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/aggFTW/proposals/labels{/name}","releases_url":"https://api.github.com/repos/aggFTW/proposals/releases{/id}","created_at":"2015-09-11T23:24:50Z","updated_at":"2015-09-04T23:48:24Z","pushed_at":"2015-09-12T03:18:11Z","git_url":"git://github.com/aggFTW/proposals.git","ssh_url":"git@github.com:aggFTW/proposals.git","clone_url":"https://github.com/aggFTW/proposals.git","svn_url":"https://github.com/aggFTW/proposals","homepage":null,"size":85,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"jupyter-incubator:master","ref":"master","sha":"2ca82277328e8d74e96d771d1be6b704a8dc1cab","user":{"login":"jupyter-incubator","id":14002361,"avatar_url":"https://avatars.githubusercontent.com/u/14002361?v=3","gravatar_id":"","url":"https://api.github.com/users/jupyter-incubator","html_url":"https://github.com/jupyter-incubator","followers_url":"https://api.github.com/users/jupyter-incubator/followers","following_url":"https://api.github.com/users/jupyter-incubator/following{/other_user}","gists_url":"https://api.github.com/users/jupyter-incubator/gists{/gist_id}","starred_url":"https://api.github.com/users/jupyter-incubator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jupyter-incubator/subscriptions","organizations_url":"https://api.github.com/users/jupyter-incubator/orgs","repos_url":"https://api.github.com/users/jupyter-incubator/repos","events_url":"https://api.github.com/users/jupyter-incubator/events{/privacy}","received_events_url":"https://api.github.com/users/jupyter-incubator/received_events","type":"Organization","site_admin":false},"repo":{"id":41942291,"name":"proposals","full_name":"jupyter-incubator/proposals","owner":{"login":"jupyter-incubator","id":14002361,"avatar_url":"https://avatars.githubusercontent.com/u/14002361?v=3","gravatar_id":"","url":"https://api.github.com/users/jupyter-incubator","html_url":"https://github.com/jupyter-incubator","followers_url":"https://api.github.com/users/jupyter-incubator/followers","following_url":"https://api.github.com/users/jupyter-incubator/following{/other_user}","gists_url":"https://api.github.com/users/jupyter-incubator/gists{/gist_id}","starred_url":"https://api.github.com/users/jupyter-incubator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jupyter-incubator/subscriptions","organizations_url":"https://api.github.com/users/jupyter-incubator/orgs","repos_url":"https://api.github.com/users/jupyter-incubator/repos","events_url":"https://api.github.com/users/jupyter-incubator/events{/privacy}","received_events_url":"https://api.github.com/users/jupyter-incubator/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/jupyter-incubator/proposals","description":"Proposals for new Jupyter subprojects to enter into incubation","fork":false,"url":"https://api.github.com/repos/jupyter-incubator/proposals","forks_url":"https://api.github.com/repos/jupyter-incubator/proposals/forks","keys_url":"https://api.github.com/repos/jupyter-incubator/proposals/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jupyter-incubator/proposals/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jupyter-incubator/proposals/teams","hooks_url":"https://api.github.com/repos/jupyter-incubator/proposals/hooks","issue_events_url":"https://api.github.com/repos/jupyter-incubator/proposals/issues/events{/number}","events_url":"https://api.github.com/repos/jupyter-incubator/proposals/events","assignees_url":"https://api.github.com/repos/jupyter-incubator/proposals/assignees{/user}","branches_url":"https://api.github.com/repos/jupyter-incubator/proposals/branches{/branch}","tags_url":"https://api.github.com/repos/jupyter-incubator/proposals/tags","blobs_url":"https://api.github.com/repos/jupyter-incubator/proposals/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jupyter-incubator/proposals/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jupyter-incubator/proposals/git/refs{/sha}","trees_url":"https://api.github.com/repos/jupyter-incubator/proposals/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jupyter-incubator/proposals/statuses/{sha}","languages_url":"https://api.github.com/repos/jupyter-incubator/proposals/languages","stargazers_url":"https://api.github.com/repos/jupyter-incubator/proposals/stargazers","contributors_url":"https://api.github.com/repos/jupyter-incubator/proposals/contributors","subscribers_url":"https://api.github.com/repos/jupyter-incubator/proposals/subscribers","subscription_url":"https://api.github.com/repos/jupyter-incubator/proposals/subscription","commits_url":"https://api.github.com/repos/jupyter-incubator/proposals/commits{/sha}","git_commits_url":"https://api.github.com/repos/jupyter-incubator/proposals/git/commits{/sha}","comments_url":"https://api.github.com/repos/jupyter-incubator/proposals/comments{/number}","issue_comment_url":"https://api.github.com/repos/jupyter-incubator/proposals/issues/comments{/number}","contents_url":"https://api.github.com/repos/jupyter-incubator/proposals/contents/{+path}","compare_url":"https://api.github.com/repos/jupyter-incubator/proposals/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jupyter-incubator/proposals/merges","archive_url":"https://api.github.com/repos/jupyter-incubator/proposals/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jupyter-incubator/proposals/downloads","issues_url":"https://api.github.com/repos/jupyter-incubator/proposals/issues{/number}","pulls_url":"https://api.github.com/repos/jupyter-incubator/proposals/pulls{/number}","milestones_url":"https://api.github.com/repos/jupyter-incubator/proposals/milestones{/number}","notifications_url":"https://api.github.com/repos/jupyter-incubator/proposals/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jupyter-incubator/proposals/labels{/name}","releases_url":"https://api.github.com/repos/jupyter-incubator/proposals/releases{/id}","created_at":"2015-09-04T23:48:24Z","updated_at":"2015-09-04T23:48:24Z","pushed_at":"2015-09-12T03:18:12Z","git_url":"git://github.com/jupyter-incubator/proposals.git","ssh_url":"git@github.com:jupyter-incubator/proposals.git","clone_url":"https://github.com/jupyter-incubator/proposals.git","svn_url":"https://github.com/jupyter-incubator/proposals","homepage":null,"size":164,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"open_issues_count":1,"forks":3,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2"},"html":{"href":"https://github.com/jupyter-incubator/proposals/pull/2"},"issue":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/issues/2"},"comments":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/issues/2/comments"},"review_comments":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2/comments"},"review_comment":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/pulls/2/commits"},"statuses":{"href":"https://api.github.com/repos/jupyter-incubator/proposals/statuses/55e445f6468c8fce714daebc58333b648e5b25f2"}}}}