{"action":"created","issue":{"url":"https://api.github.com/repos/akka/akka/issues/17341","labels_url":"https://api.github.com/repos/akka/akka/issues/17341/labels{/name}","comments_url":"https://api.github.com/repos/akka/akka/issues/17341/comments","events_url":"https://api.github.com/repos/akka/akka/issues/17341/events","html_url":"https://github.com/akka/akka/issues/17341","id":72046218,"number":17341,"title":"Actor starvation when self-messages and different dispatchers are involved","user":{"login":"clockfly","id":2595532,"avatar_url":"https://avatars.githubusercontent.com/u/2595532?v=3","gravatar_id":"","url":"https://api.github.com/users/clockfly","html_url":"https://github.com/clockfly","followers_url":"https://api.github.com/users/clockfly/followers","following_url":"https://api.github.com/users/clockfly/following{/other_user}","gists_url":"https://api.github.com/users/clockfly/gists{/gist_id}","starred_url":"https://api.github.com/users/clockfly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clockfly/subscriptions","organizations_url":"https://api.github.com/users/clockfly/orgs","repos_url":"https://api.github.com/users/clockfly/repos","events_url":"https://api.github.com/users/clockfly/events{/privacy}","received_events_url":"https://api.github.com/users/clockfly/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/akka/akka/labels/3%20-%20in%20progress","name":"3 - in progress","color":"eb6420"},{"url":"https://api.github.com/repos/akka/akka/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/akka/akka/labels/t:core","name":"t:core","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"drewhk","id":932285,"avatar_url":"https://avatars.githubusercontent.com/u/932285?v=3","gravatar_id":"","url":"https://api.github.com/users/drewhk","html_url":"https://github.com/drewhk","followers_url":"https://api.github.com/users/drewhk/followers","following_url":"https://api.github.com/users/drewhk/following{/other_user}","gists_url":"https://api.github.com/users/drewhk/gists{/gist_id}","starred_url":"https://api.github.com/users/drewhk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drewhk/subscriptions","organizations_url":"https://api.github.com/users/drewhk/orgs","repos_url":"https://api.github.com/users/drewhk/repos","events_url":"https://api.github.com/users/drewhk/events{/privacy}","received_events_url":"https://api.github.com/users/drewhk/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/akka/akka/milestones/47","html_url":"https://github.com/akka/akka/milestones/2.3.11","labels_url":"https://api.github.com/repos/akka/akka/milestones/47/labels","id":1098190,"number":47,"title":"2.3.11","description":"","creator":{"login":"rkuhn","id":470469,"avatar_url":"https://avatars.githubusercontent.com/u/470469?v=3","gravatar_id":"","url":"https://api.github.com/users/rkuhn","html_url":"https://github.com/rkuhn","followers_url":"https://api.github.com/users/rkuhn/followers","following_url":"https://api.github.com/users/rkuhn/following{/other_user}","gists_url":"https://api.github.com/users/rkuhn/gists{/gist_id}","starred_url":"https://api.github.com/users/rkuhn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkuhn/subscriptions","organizations_url":"https://api.github.com/users/rkuhn/orgs","repos_url":"https://api.github.com/users/rkuhn/repos","events_url":"https://api.github.com/users/rkuhn/events{/privacy}","received_events_url":"https://api.github.com/users/rkuhn/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":1,"state":"open","created_at":"2015-05-06T08:01:44Z","updated_at":"2015-05-06T10:58:29Z","due_on":null,"closed_at":null},"comments":22,"created_at":"2015-04-30T02:57:11Z","updated_at":"2015-05-06T15:06:21Z","closed_at":null,"body":"## Problem Statement:\r\n\r\nWe have a benchmark test which process 10 million message per second(project https://github.com/intel-hadoop/gearpump), during the test, I found sometimes some message cannot be delivered to target actor.\r\n\r\n## Test case description:\r\nThe test is like this:\r\nI have 600 upstream actor, and 600 downstream actors distribued in 60 JVM. the upstream actors will crazily send data to downstream actors in a shuffle way. The upstream actors rely on self-sending message to trigger the repeation of sending. \r\n\r\nupstream tasks and downstream tasks will share a custom dispatcher(not default dispatcher, or remote-default-dispatcher).\r\n\r\nUpstream Actor logic:\r\n1. upstream actor A send itself a tick message\r\n2. in receive of A, sending one messge to downstream actors. After sending, send itself another tick message\r\n3. repeat step 2\r\n\r\n## Trouble-shooting and findings\r\nI compiled akka source to add log trace, and confirmed that the lost message is already inserted into target Mailbox queue(SingleConsumerOnlyUnboundedMailbox), and the Malibox(Runnable) is already inserted into internal task queue of fork-join pool. **The strange part** is that the mbox was **NOT** get scheduled to a real thead and run.\r\n\r\n### Finding1: about akka Version\r\nI find version akka 2.3.6 don't have this problem, while version 2.3.9 has.\r\n\r\n### Finding2: about self-sending message\r\nFor my test, the upstream actors need to send themselves repeatedly tick message so that they can trigger themselves to send more message downstream. If I change the self-sending message, to be an indirect self sending, then the problem is gone. \r\nFor example, instead of a sending directly to itself A->A, I create a intermediate actor C, A->C->A,\r\n\r\n### Finding 3: About changes in latest akka version\r\nSince 2.3.6 doesn't have this problem, I browser through the akka history, and find a suspicious change. https://github.com/akka/akka/pull/16152/files#diff-2c9d0638ce3a1f007623ac9d1ead1725R379\r\n\r\nThe intent of this PR is to make MailBox a subclass of ForkjoinTask, so that we can optimize the performance by fully using fork-join ideas. I doubt the changes in this PR will also create fairness problem. \r\n\r\nHere is some change in AkkaForkJoinPool\r\n```scala\r\n     extends ForkJoinPool(parallelism, threadFactory, unhandledExceptionHandler, true) with LoadMetrics {\r\n-    override def execute(r: Runnable): Unit =\t\t+    override def execute(r: Runnable): Unit = {\r\n-      if (r eq null) throw new NullPointerException else super.execute(new AkkaForkJoinTask(r))\t\t+      if (r eq null) throw new NullPointerException(\"The Runnable must not be null\")\r\n+      val task =\r\n+        if (r.isInstanceOf[ForkJoinTask[_]]) r.asInstanceOf[ForkJoinTask[Any]]\r\n+        else new AkkaForkJoinTask(r)\r\n+      Thread.currentThread match {\r\n+        case worker: ForkJoinWorkerThread if worker.getPool eq this ⇒ task.fork()\r\n+        case _ ⇒ super.execute(task)\r\n+      }\r\n```\r\n\r\nIf actor A is sending itself message, then it will trigger AkkaForkJoinPool.execute, as the calling thread is already in the fork-join pool, \r\n```scala\r\n+      Thread.currentThread match {\r\n+        case worker: ForkJoinWorkerThread if worker.getPool eq this ⇒ task.fork()\r\n+        case _ ⇒ super.execute(task)\r\n+      }\r\n```\r\n\r\nSo it will pick first branch which execute ```task.fork()```\r\n\r\nI doubt task.fork() will have higher priority than the other branch \"super.execute(task)\", so possibly the later will not get scheduled.\r\n\r\n\r\n\r\nThanks in advance.\r\nSean\r\n"},"comment":{"url":"https://api.github.com/repos/akka/akka/issues/comments/99505418","html_url":"https://github.com/akka/akka/issues/17341#issuecomment-99505418","issue_url":"https://api.github.com/repos/akka/akka/issues/17341","id":99505418,"user":{"login":"drewhk","id":932285,"avatar_url":"https://avatars.githubusercontent.com/u/932285?v=3","gravatar_id":"","url":"https://api.github.com/users/drewhk","html_url":"https://github.com/drewhk","followers_url":"https://api.github.com/users/drewhk/followers","following_url":"https://api.github.com/users/drewhk/following{/other_user}","gists_url":"https://api.github.com/users/drewhk/gists{/gist_id}","starred_url":"https://api.github.com/users/drewhk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drewhk/subscriptions","organizations_url":"https://api.github.com/users/drewhk/orgs","repos_url":"https://api.github.com/users/drewhk/repos","events_url":"https://api.github.com/users/drewhk/events{/privacy}","received_events_url":"https://api.github.com/users/drewhk/received_events","type":"User","site_admin":false},"created_at":"2015-05-06T15:06:21Z","updated_at":"2015-05-06T15:06:21Z","body":"If everyone agrees, I will prepare a PR that reverts to 2.3.8 implementation and add reproducer tests for the starvation scenarios so we don't make the same mistakes in the future."}}