{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/42831391","id":42831391,"diff_hunk":"@@ -127,37 +170,28 @@ private[netty] class NettyRpcEnv(\n           val ack = response.asInstanceOf[Ack]\n           logTrace(s\"Received ack from ${ack.sender}\")\n         case Failure(e) =>\n-          logError(s\"Exception when sending $message\", e)\n+          logWarning(s\"Exception when sending $message\", e)\n       }(ThreadUtils.sameThread)\n     } else {\n       // Message to a remote RPC endpoint.\n-      try {\n-        // `createClient` will block if it cannot find a known connection, so we should run it in\n-        // clientConnectionExecutor\n-        clientConnectionExecutor.execute(new Runnable {\n-          override def run(): Unit = Utils.tryLogNonFatalError {\n-            val client = clientFactory.createClient(remoteAddr.host, remoteAddr.port)\n-            client.sendRpc(serialize(message), new RpcResponseCallback {\n-\n-              override def onFailure(e: Throwable): Unit = {\n-                logError(s\"Exception when sending $message\", e)\n-              }\n-\n-              override def onSuccess(response: Array[Byte]): Unit = {\n-                val ack = deserialize[Ack](response)\n-                logDebug(s\"Receive ack from ${ack.sender}\")\n-              }\n-            })\n-          }\n-        })\n-      } catch {\n-        case e: RejectedExecutionException =>\n-          // `send` after shutting clientConnectionExecutor down, ignore it\n-          logWarning(s\"Cannot send $message because RpcEnv is stopped\")\n-      }\n+      postToOutbox(remoteAddr, OutboxMessage(serialize(message), new RpcResponseCallback {\n+\n+        override def onFailure(e: Throwable): Unit = {\n+          logWarning(s\"Exception when sending $message\", e)\n+        }\n+\n+        override def onSuccess(response: Array[Byte]): Unit = {\n+          val ack = deserialize[Ack](response)\n+          logDebug(s\"Receive ack from ${ack.sender}\")","path":"core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala","position":112,"original_position":112,"commit_id":"a5def400a54ac99e004dbf82764a36c9af092248","original_commit_id":"a5def400a54ac99e004dbf82764a36c9af092248","user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"body":"is this log useful? maybe we don't need to do much here ...\r\n","created_at":"2015-10-23T03:57:42Z","updated_at":"2015-10-23T03:57:42Z","html_url":"https://github.com/apache/spark/pull/9197#discussion_r42831391","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/9197","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/42831391"},"html":{"href":"https://github.com/apache/spark/pull/9197#discussion_r42831391"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/9197"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9197","id":48317699,"html_url":"https://github.com/apache/spark/pull/9197","diff_url":"https://github.com/apache/spark/pull/9197.diff","patch_url":"https://github.com/apache/spark/pull/9197.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/9197","number":9197,"state":"open","locked":false,"title":"[SPARK-11098][Core]Add Outbox to cache the sending messages to resolve the message disorder issue","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"body":"The current NettyRpc has a message order issue because it uses a thread pool to send messages. E.g., running the following two lines in the same thread,\r\n\r\n```\r\nref.send(\"A\")\r\nref.send(\"B\")\r\n```\r\n\r\nThe remote endpoint may see \"B\" before \"A\" because sending \"A\" and \"B\" are in parallel.\r\nTo resolve this issue, this PR added an outbox for each connection, and if we are connecting to the remote node when sending messages, just cache the sending messages in the outbox and send them one by one when the connection is established.","created_at":"2015-10-21T11:19:24Z","updated_at":"2015-10-23T03:57:42Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f70137a5aa9cdef90744e8c11cd746b920c7b31c","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/9197/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/9197/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/9197/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/a5def400a54ac99e004dbf82764a36c9af092248","head":{"label":"zsxwing:rpc-outbox","ref":"rpc-outbox","sha":"a5def400a54ac99e004dbf82764a36c9af092248","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"repo":{"id":17894272,"name":"spark","full_name":"zsxwing/spark","owner":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/zsxwing/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/zsxwing/spark","forks_url":"https://api.github.com/repos/zsxwing/spark/forks","keys_url":"https://api.github.com/repos/zsxwing/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zsxwing/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zsxwing/spark/teams","hooks_url":"https://api.github.com/repos/zsxwing/spark/hooks","issue_events_url":"https://api.github.com/repos/zsxwing/spark/issues/events{/number}","events_url":"https://api.github.com/repos/zsxwing/spark/events","assignees_url":"https://api.github.com/repos/zsxwing/spark/assignees{/user}","branches_url":"https://api.github.com/repos/zsxwing/spark/branches{/branch}","tags_url":"https://api.github.com/repos/zsxwing/spark/tags","blobs_url":"https://api.github.com/repos/zsxwing/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zsxwing/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zsxwing/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/zsxwing/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zsxwing/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/zsxwing/spark/languages","stargazers_url":"https://api.github.com/repos/zsxwing/spark/stargazers","contributors_url":"https://api.github.com/repos/zsxwing/spark/contributors","subscribers_url":"https://api.github.com/repos/zsxwing/spark/subscribers","subscription_url":"https://api.github.com/repos/zsxwing/spark/subscription","commits_url":"https://api.github.com/repos/zsxwing/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/zsxwing/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/zsxwing/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/zsxwing/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/zsxwing/spark/contents/{+path}","compare_url":"https://api.github.com/repos/zsxwing/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zsxwing/spark/merges","archive_url":"https://api.github.com/repos/zsxwing/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zsxwing/spark/downloads","issues_url":"https://api.github.com/repos/zsxwing/spark/issues{/number}","pulls_url":"https://api.github.com/repos/zsxwing/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/zsxwing/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/zsxwing/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zsxwing/spark/labels{/name}","releases_url":"https://api.github.com/repos/zsxwing/spark/releases{/id}","created_at":"2014-03-19T06:21:04Z","updated_at":"2015-03-09T07:52:42Z","pushed_at":"2015-10-23T02:55:13Z","git_url":"git://github.com/zsxwing/spark.git","ssh_url":"git@github.com:zsxwing/spark.git","clone_url":"https://github.com/zsxwing/spark.git","svn_url":"https://github.com/zsxwing/spark","homepage":null,"size":121355,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"5cdea7d1e54e751f928876ead33fc01e7e2f3437","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-10-23T02:31:38Z","pushed_at":"2015-10-23T02:59:16Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1643372,"stargazers_count":5845,"watchers_count":5845,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5458,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":333,"forks":5458,"open_issues":333,"watchers":5845,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/9197"},"html":{"href":"https://github.com/apache/spark/pull/9197"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/9197"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/9197/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/9197/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/9197/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/a5def400a54ac99e004dbf82764a36c9af092248"}}}}