{"action":"created","comment":{"url":"https://api.github.com/repos/rxin/spark/pulls/comments/40583777","id":40583777,"diff_hunk":"@@ -0,0 +1,91 @@\n+package org.apache.spark.sql.streamv1\n+\n+import org.apache.spark.sql.types.StructType\n+import org.apache.spark.sql.{Column, DataFrame}\n+\n+\n+/**\n+ * Version A: A single StreamFrame abstraction to represent unwindowed stream and windowed stream.\n+ *\n+ * There are two alternatives for the blocking operations in StreamFrame when it is not windowed:\n+ *\n+ * A1. Blocking operations return new StreamFrames, and emits a new tuple for every update.\n+ *     As an example, sf.groupby(\"id\").count() will emit a tuple every time we see a new record\n+ *     for \"id\", i.e. a running count. Note that these operations will be expensive, because\n+ *     they require ordering all the inputs by time.","path":"sql/core/src/main/scala/org/apache/spark/sql/streamv1/StreamFrame.scala","position":15,"original_position":15,"commit_id":"75060e687b11b7bb6868e388b15808fe99991a63","original_commit_id":"75060e687b11b7bb6868e388b15808fe99991a63","user":{"login":"mateiz","id":228859,"avatar_url":"https://avatars.githubusercontent.com/u/228859?v=3","gravatar_id":"","url":"https://api.github.com/users/mateiz","html_url":"https://github.com/mateiz","followers_url":"https://api.github.com/users/mateiz/followers","following_url":"https://api.github.com/users/mateiz/following{/other_user}","gists_url":"https://api.github.com/users/mateiz/gists{/gist_id}","starred_url":"https://api.github.com/users/mateiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mateiz/subscriptions","organizations_url":"https://api.github.com/users/mateiz/orgs","repos_url":"https://api.github.com/users/mateiz/repos","events_url":"https://api.github.com/users/mateiz/events{/privacy}","received_events_url":"https://api.github.com/users/mateiz/received_events","type":"User","site_admin":false},"body":"What does \"emitting\" a tuple mean? In other words, how does this API tie in with event time and stuff like that?\r\n\r\nGenerally, the way I understood event time is that you want to define a query based on the event time in the data, and then evaluate it at possibly multiple points in \"real\" (processing) time to get a different answer for the same event time window.\r\n\r\nIf you mean that this kind of stream is the same as one with an infinite window, that does make sense.","created_at":"2015-09-28T18:00:29Z","updated_at":"2015-09-28T18:00:29Z","html_url":"https://github.com/rxin/spark/pull/17#discussion_r40583777","pull_request_url":"https://api.github.com/repos/rxin/spark/pulls/17","_links":{"self":{"href":"https://api.github.com/repos/rxin/spark/pulls/comments/40583777"},"html":{"href":"https://github.com/rxin/spark/pull/17#discussion_r40583777"},"pull_request":{"href":"https://api.github.com/repos/rxin/spark/pulls/17"}}},"pull_request":{"url":"https://api.github.com/repos/rxin/spark/pulls/17","id":45947353,"html_url":"https://github.com/rxin/spark/pull/17","diff_url":"https://github.com/rxin/spark/pull/17.diff","patch_url":"https://github.com/rxin/spark/pull/17.patch","issue_url":"https://api.github.com/repos/rxin/spark/issues/17","number":17,"state":"open","locked":false,"title":"StreamFrame API alternatives.","user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"body":"This pull request proposes 3 alternatives for StreamFrame API.\r\n\r\n- __A1__: A single StreamFrame abstraction for both non-windowed and windowed stream. Blocking operations work also on non-windowed streams, as they just return a new stream with a new tuple emitted for every update (i.e. can be used to query partial results).\r\n\r\n- __A2__: A single StreamFrame abstraction for both non-windowed and windowed stream. Blocking operations throw runtime exceptions on non-windowed stream. This is similar to the PCollection idea in Google's Cloud Dataflow.\r\n\r\n- __B__: Two abstractions: StreamFrame and WindowedStreamFrame. Blocking operations are only available on WindowedStreamFrame.\r\n\r\nAt this point, we are mostly looking for feedback about the high level alternatives (e.g. not the detailed function names, window specs, etc).\r\n ","created_at":"2015-09-24T20:43:55Z","updated_at":"2015-09-28T18:00:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9fceada19869e20d274555710668ca590a6f9f37","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/rxin/spark/pulls/17/commits","review_comments_url":"https://api.github.com/repos/rxin/spark/pulls/17/comments","review_comment_url":"https://api.github.com/repos/rxin/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/rxin/spark/issues/17/comments","statuses_url":"https://api.github.com/repos/rxin/spark/statuses/75060e687b11b7bb6868e388b15808fe99991a63","head":{"label":"rxin:streamframe-api","ref":"streamframe-api","sha":"75060e687b11b7bb6868e388b15808fe99991a63","user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"repo":{"id":17192644,"name":"spark","full_name":"rxin/spark","owner":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/rxin/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/rxin/spark","forks_url":"https://api.github.com/repos/rxin/spark/forks","keys_url":"https://api.github.com/repos/rxin/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rxin/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rxin/spark/teams","hooks_url":"https://api.github.com/repos/rxin/spark/hooks","issue_events_url":"https://api.github.com/repos/rxin/spark/issues/events{/number}","events_url":"https://api.github.com/repos/rxin/spark/events","assignees_url":"https://api.github.com/repos/rxin/spark/assignees{/user}","branches_url":"https://api.github.com/repos/rxin/spark/branches{/branch}","tags_url":"https://api.github.com/repos/rxin/spark/tags","blobs_url":"https://api.github.com/repos/rxin/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rxin/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rxin/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/rxin/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rxin/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/rxin/spark/languages","stargazers_url":"https://api.github.com/repos/rxin/spark/stargazers","contributors_url":"https://api.github.com/repos/rxin/spark/contributors","subscribers_url":"https://api.github.com/repos/rxin/spark/subscribers","subscription_url":"https://api.github.com/repos/rxin/spark/subscription","commits_url":"https://api.github.com/repos/rxin/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/rxin/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/rxin/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/rxin/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/rxin/spark/contents/{+path}","compare_url":"https://api.github.com/repos/rxin/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rxin/spark/merges","archive_url":"https://api.github.com/repos/rxin/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rxin/spark/downloads","issues_url":"https://api.github.com/repos/rxin/spark/issues{/number}","pulls_url":"https://api.github.com/repos/rxin/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/rxin/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/rxin/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rxin/spark/labels{/name}","releases_url":"https://api.github.com/repos/rxin/spark/releases{/id}","created_at":"2014-02-25T23:55:52Z","updated_at":"2015-07-10T06:38:54Z","pushed_at":"2015-09-24T20:59:52Z","git_url":"git://github.com/rxin/spark.git","ssh_url":"git@github.com:rxin/spark.git","clone_url":"https://github.com/rxin/spark.git","svn_url":"https://github.com/rxin/spark","homepage":null,"size":127156,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":1,"forks":5,"open_issues":1,"watchers":1,"default_branch":"master"}},"base":{"label":"rxin:master","ref":"master","sha":"d91967e159f416924bbd7f0db25156588d4bd7b1","user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"repo":{"id":17192644,"name":"spark","full_name":"rxin/spark","owner":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/rxin/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/rxin/spark","forks_url":"https://api.github.com/repos/rxin/spark/forks","keys_url":"https://api.github.com/repos/rxin/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rxin/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rxin/spark/teams","hooks_url":"https://api.github.com/repos/rxin/spark/hooks","issue_events_url":"https://api.github.com/repos/rxin/spark/issues/events{/number}","events_url":"https://api.github.com/repos/rxin/spark/events","assignees_url":"https://api.github.com/repos/rxin/spark/assignees{/user}","branches_url":"https://api.github.com/repos/rxin/spark/branches{/branch}","tags_url":"https://api.github.com/repos/rxin/spark/tags","blobs_url":"https://api.github.com/repos/rxin/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rxin/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rxin/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/rxin/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rxin/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/rxin/spark/languages","stargazers_url":"https://api.github.com/repos/rxin/spark/stargazers","contributors_url":"https://api.github.com/repos/rxin/spark/contributors","subscribers_url":"https://api.github.com/repos/rxin/spark/subscribers","subscription_url":"https://api.github.com/repos/rxin/spark/subscription","commits_url":"https://api.github.com/repos/rxin/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/rxin/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/rxin/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/rxin/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/rxin/spark/contents/{+path}","compare_url":"https://api.github.com/repos/rxin/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rxin/spark/merges","archive_url":"https://api.github.com/repos/rxin/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rxin/spark/downloads","issues_url":"https://api.github.com/repos/rxin/spark/issues{/number}","pulls_url":"https://api.github.com/repos/rxin/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/rxin/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/rxin/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rxin/spark/labels{/name}","releases_url":"https://api.github.com/repos/rxin/spark/releases{/id}","created_at":"2014-02-25T23:55:52Z","updated_at":"2015-07-10T06:38:54Z","pushed_at":"2015-09-24T20:59:52Z","git_url":"git://github.com/rxin/spark.git","ssh_url":"git@github.com:rxin/spark.git","clone_url":"https://github.com/rxin/spark.git","svn_url":"https://github.com/rxin/spark","homepage":null,"size":127156,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":1,"forks":5,"open_issues":1,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/rxin/spark/pulls/17"},"html":{"href":"https://github.com/rxin/spark/pull/17"},"issue":{"href":"https://api.github.com/repos/rxin/spark/issues/17"},"comments":{"href":"https://api.github.com/repos/rxin/spark/issues/17/comments"},"review_comments":{"href":"https://api.github.com/repos/rxin/spark/pulls/17/comments"},"review_comment":{"href":"https://api.github.com/repos/rxin/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/rxin/spark/pulls/17/commits"},"statuses":{"href":"https://api.github.com/repos/rxin/spark/statuses/75060e687b11b7bb6868e388b15808fe99991a63"}}}}