{"action":"created","issue":{"url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5851","labels_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5851/labels{/name}","comments_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5851/comments","events_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5851/events","html_url":"https://github.com/rapid7/metasploit-framework/pull/5851","id":101176587,"number":5851,"title":"Improve meterpreter packet dispatch performance","user":{"login":"joncave","id":621103,"avatar_url":"https://avatars.githubusercontent.com/u/621103?v=3","gravatar_id":"","url":"https://api.github.com/users/joncave","html_url":"https://github.com/joncave","followers_url":"https://api.github.com/users/joncave/followers","following_url":"https://api.github.com/users/joncave/following{/other_user}","gists_url":"https://api.github.com/users/joncave/gists{/gist_id}","starred_url":"https://api.github.com/users/joncave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joncave/subscriptions","organizations_url":"https://api.github.com/users/joncave/orgs","repos_url":"https://api.github.com/users/joncave/repos","events_url":"https://api.github.com/users/joncave/events{/privacy}","received_events_url":"https://api.github.com/users/joncave/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rapid7/metasploit-framework/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/rapid7/metasploit-framework/labels/library","name":"library","color":"fad8c7"},{"url":"https://api.github.com/repos/rapid7/metasploit-framework/labels/meterpreter","name":"meterpreter","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"jlee-r7","id":1173675,"avatar_url":"https://avatars.githubusercontent.com/u/1173675?v=3","gravatar_id":"","url":"https://api.github.com/users/jlee-r7","html_url":"https://github.com/jlee-r7","followers_url":"https://api.github.com/users/jlee-r7/followers","following_url":"https://api.github.com/users/jlee-r7/following{/other_user}","gists_url":"https://api.github.com/users/jlee-r7/gists{/gist_id}","starred_url":"https://api.github.com/users/jlee-r7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jlee-r7/subscriptions","organizations_url":"https://api.github.com/users/jlee-r7/orgs","repos_url":"https://api.github.com/users/jlee-r7/repos","events_url":"https://api.github.com/users/jlee-r7/events{/privacy}","received_events_url":"https://api.github.com/users/jlee-r7/received_events","type":"User","site_admin":false},"milestone":null,"comments":19,"created_at":"2015-08-15T13:34:26Z","updated_at":"2015-11-16T18:28:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/rapid7/metasploit-framework/pulls/5851","html_url":"https://github.com/rapid7/metasploit-framework/pull/5851","diff_url":"https://github.com/rapid7/metasploit-framework/pull/5851.diff","patch_url":"https://github.com/rapid7/metasploit-framework/pull/5851.patch"},"body":"This PR attempts to fix #5845 by switching away from select() based sleeps instead of reducing the sleep times as in #5849. The fix is in three parts: use Queue.pop() in the dispatch thread, use a Mutex/CV in the packet waiter, and a bug fix in the waiter notification.\r\n\r\nSwitching `pqueue` to a `Queue` allows pop to be used to wait for a packet. Until a packet is pushed into the queue [the thread is suspended](http://ruby-doc.org/core-2.1.6/Queue.html#method-i-pop). My testing shows that this is more efficient than the current approach of sleeping and polling the queue length.\r\n\r\nSimilarly I switched the packet response waiter back to a Mutex/CV based solution. This differs from the original implementation in that I make use of the `wait()` method's timeout parameter instead of a separate use of the `Timeout` class. Hopefully this solves the race which caused the change to the current select code (5757685b5904e473acb5b4a1bcd2adaf9090203b). However, I'm not sure what that issue actually was so I can't be certain.\r\n\r\nFinally I noticed that the `backlog == incomplete` delay was being triggered very often, even for a file that's read in a single packet. It turned out that `dispatch_inbound_packet()` would check the return value of `notify_response_waiter()` to see if the response had been handled. If the loop is broken early due to a waiter being notified `nil` is returned leading to the dispatch thread believing that the packet wasn't handled. It then goes into the incomplete list and goes around again.\r\n\r\nIn my tests all of these changes mean that a 32M download goes from 52s to 0.5s on localhost and from 65s to 15s over the internet. I also didn't notice any significant increase in CPU usage with 30 sessions connected."},"comment":{"url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/comments/157128158","html_url":"https://github.com/rapid7/metasploit-framework/pull/5851#issuecomment-157128158","issue_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5851","id":157128158,"user":{"login":"joncave","id":621103,"avatar_url":"https://avatars.githubusercontent.com/u/621103?v=3","gravatar_id":"","url":"https://api.github.com/users/joncave","html_url":"https://github.com/joncave","followers_url":"https://api.github.com/users/joncave/followers","following_url":"https://api.github.com/users/joncave/following{/other_user}","gists_url":"https://api.github.com/users/joncave/gists{/gist_id}","starred_url":"https://api.github.com/users/joncave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joncave/subscriptions","organizations_url":"https://api.github.com/users/joncave/orgs","repos_url":"https://api.github.com/users/joncave/repos","events_url":"https://api.github.com/users/joncave/events{/privacy}","received_events_url":"https://api.github.com/users/joncave/received_events","type":"User","site_admin":false},"created_at":"2015-11-16T18:28:12Z","updated_at":"2015-11-16T18:28:12Z","body":"@jlee-r7 Ah, sorry, I misread (missed part about timing out of the waiters)."}}