{"action":"created","issue":{"url":"https://api.github.com/repos/akka/akka/issues/17216","labels_url":"https://api.github.com/repos/akka/akka/issues/17216/labels{/name}","comments_url":"https://api.github.com/repos/akka/akka/issues/17216/comments","events_url":"https://api.github.com/repos/akka/akka/issues/17216/events","html_url":"https://github.com/akka/akka/issues/17216","id":69071198,"number":17216,"title":"fix open-coded spinlock in DirectByteBufferPool","user":{"login":"rkuhn","id":470469,"avatar_url":"https://avatars.githubusercontent.com/u/470469?v=3","gravatar_id":"","url":"https://api.github.com/users/rkuhn","html_url":"https://github.com/rkuhn","followers_url":"https://api.github.com/users/rkuhn/followers","following_url":"https://api.github.com/users/rkuhn/following{/other_user}","gists_url":"https://api.github.com/users/rkuhn/gists{/gist_id}","starred_url":"https://api.github.com/users/rkuhn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkuhn/subscriptions","organizations_url":"https://api.github.com/users/rkuhn/orgs","repos_url":"https://api.github.com/users/rkuhn/repos","events_url":"https://api.github.com/users/rkuhn/events{/privacy}","received_events_url":"https://api.github.com/users/rkuhn/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/akka/akka/labels/1%20-%20triaged","name":"1 - triaged","color":"5319e7"},{"url":"https://api.github.com/repos/akka/akka/labels/t:core","name":"t:core","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/akka/akka/milestones/40","html_url":"https://github.com/akka/akka/milestones/2.3.10","labels_url":"https://api.github.com/repos/akka/akka/milestones/40/labels","id":1047150,"number":40,"title":"2.3.10","description":null,"creator":{"login":"ktoso","id":120979,"avatar_url":"https://avatars.githubusercontent.com/u/120979?v=3","gravatar_id":"","url":"https://api.github.com/users/ktoso","html_url":"https://github.com/ktoso","followers_url":"https://api.github.com/users/ktoso/followers","following_url":"https://api.github.com/users/ktoso/following{/other_user}","gists_url":"https://api.github.com/users/ktoso/gists{/gist_id}","starred_url":"https://api.github.com/users/ktoso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktoso/subscriptions","organizations_url":"https://api.github.com/users/ktoso/orgs","repos_url":"https://api.github.com/users/ktoso/repos","events_url":"https://api.github.com/users/ktoso/events{/privacy}","received_events_url":"https://api.github.com/users/ktoso/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":21,"state":"open","created_at":"2015-03-31T15:01:34Z","updated_at":"2015-04-17T06:24:22Z","due_on":null,"closed_at":null},"comments":17,"created_at":"2015-04-17T06:17:16Z","updated_at":"2015-04-17T09:41:45Z","closed_at":null,"body":"from Akara on the mailing list:\r\n---\r\nScala 2.11.6, Akka 2.3.9, Spray 1.3.2.\r\n\r\nWhen testing a particular workload, we ran into CPU spikes. The avg CPU load is less than 5% but about 6-7 minutes into the run, we start to see CPU spiking to near 100% lasting for several seconds. This repeats itself every 6-7 minutes. We can't correlate this with a particular GC activity just yet. We tested against Java 7_45 but tried Java 8 without much change.\r\n\r\nSpray is configured on the default-dispatcher (Spray default).\r\n\r\nTrying to capture thread dumps at the time spikes happening reveal 9 threads spinning on the followings:\r\n~~~\r\n\"default-akka.actor.default-dispatcher-11\" prio=10 tid=0x0000000001b2b800 nid=0x3752 runnable [0x00007f1191d40000]\r\n   java.lang.Thread.State: RUNNABLE\r\n\tat akka.io.DirectByteBufferPool.takeBufferFromPool(DirectByteBufferPool.scala:41)\r\n\tat akka.io.DirectByteBufferPool.acquire(DirectByteBufferPool.scala:31)\r\n\tat akka.io.TcpConnection.doRead(TcpConnection.scala:231)\r\n\tat akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(TcpConnection.scala:87)\r\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\r\n\tat akka.actor.Actor$class.aroundReceive(Actor.scala:465)\r\n\tat akka.io.TcpConnection.aroundReceive(TcpConnection.scala:28)\r\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)\r\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:487)\r\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:221)\r\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:231)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\n~~~\r\n2 threads show this spinning:\r\n~~~\r\n\"default-akka.actor.default-dispatcher-8\" prio=10 tid=0x0000000001d6f000 nid=0x374f runnable [0x00007f11928d2000]\r\n   java.lang.Thread.State: RUNNABLE\r\n\tat akka.io.DirectByteBufferPool.offerBufferToPool(DirectByteBufferPool.scala:66)\r\n\tat akka.io.DirectByteBufferPool.release(DirectByteBufferPool.scala:34)\r\n\tat akka.io.TcpConnection.doRead(TcpConnection.scala:245)\r\n\tat akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(TcpConnection.scala:87)\r\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\r\n\tat akka.actor.Actor$class.aroundReceive(Actor.scala:465)\r\n\tat akka.io.TcpConnection.aroundReceive(TcpConnection.scala:28)\r\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)\r\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:487)\r\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:221)\r\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:231)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\n~~~\r\n1 thread is on I/O:\r\n\r\n~~~\r\n\"default-akka.actor.default-dispatcher-4\" prio=10 tid=0x0000000001be2800 nid=0x36ff runnable [0x00007f11924b0000]\r\n   java.lang.Thread.State: RUNNABLE\r\n\tat sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)\r\n\t- locked <0x00000007d3b7e940> (a java.lang.Object)\r\n\tat akka.io.TcpConnection.innerRead$1(TcpConnection.scala:216)\r\n\tat akka.io.TcpConnection.doRead(TcpConnection.scala:232)\r\n\tat akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(TcpConnection.scala:87)\r\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\r\n\tat akka.actor.Actor$class.aroundReceive(Actor.scala:465)\r\n\tat akka.io.TcpConnection.aroundReceive(TcpConnection.scala:28)\r\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)\r\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:487)\r\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:221)\r\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:231)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\n~~~\r\n\r\nAll in all, during this time there is no app logic being executed on the default dispatcher at all (12 threads on the default-dispatcher)."},"comment":{"url":"https://api.github.com/repos/akka/akka/issues/comments/93953638","html_url":"https://github.com/akka/akka/issues/17216#issuecomment-93953638","issue_url":"https://api.github.com/repos/akka/akka/issues/17216","id":93953638,"user":{"login":"sirthias","id":135347,"avatar_url":"https://avatars.githubusercontent.com/u/135347?v=3","gravatar_id":"","url":"https://api.github.com/users/sirthias","html_url":"https://github.com/sirthias","followers_url":"https://api.github.com/users/sirthias/followers","following_url":"https://api.github.com/users/sirthias/following{/other_user}","gists_url":"https://api.github.com/users/sirthias/gists{/gist_id}","starred_url":"https://api.github.com/users/sirthias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sirthias/subscriptions","organizations_url":"https://api.github.com/users/sirthias/orgs","repos_url":"https://api.github.com/users/sirthias/repos","events_url":"https://api.github.com/users/sirthias/events{/privacy}","received_events_url":"https://api.github.com/users/sirthias/received_events","type":"User","site_admin":false},"created_at":"2015-04-17T09:41:45Z","updated_at":"2015-04-17T09:41:45Z","body":"Ok, thank you, @jrudolph and @rkuhn!\r\n\r\nSince this is not the only spinlock that I have introduced into Akka we should also convert the other ones.\r\nThere is at least this one here:\r\nhttps://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/pattern/AskSupport.scala#L266\r\n\r\nAlso, there is one in my current code for the HTTP connection pool, which I'll change as well."}}