{"action":"created","issue":{"url":"https://api.github.com/repos/apache/storm/issues/700","labels_url":"https://api.github.com/repos/apache/storm/issues/700/labels{/name}","comments_url":"https://api.github.com/repos/apache/storm/issues/700/comments","events_url":"https://api.github.com/repos/apache/storm/issues/700/events","html_url":"https://github.com/apache/storm/pull/700","id":103309317,"number":700,"title":"[STORM-886] Automatic Back Pressure (ABP)","user":{"login":"zhuoliu","id":11683054,"avatar_url":"https://avatars.githubusercontent.com/u/11683054?v=3","gravatar_id":"","url":"https://api.github.com/users/zhuoliu","html_url":"https://github.com/zhuoliu","followers_url":"https://api.github.com/users/zhuoliu/followers","following_url":"https://api.github.com/users/zhuoliu/following{/other_user}","gists_url":"https://api.github.com/users/zhuoliu/gists{/gist_id}","starred_url":"https://api.github.com/users/zhuoliu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhuoliu/subscriptions","organizations_url":"https://api.github.com/users/zhuoliu/orgs","repos_url":"https://api.github.com/users/zhuoliu/repos","events_url":"https://api.github.com/users/zhuoliu/events{/privacy}","received_events_url":"https://api.github.com/users/zhuoliu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-08-26T16:18:51Z","updated_at":"2015-10-09T07:33:40Z","closed_at":"2015-09-18T19:30:12Z","pull_request":{"url":"https://api.github.com/repos/apache/storm/pulls/700","html_url":"https://github.com/apache/storm/pull/700","diff_url":"https://github.com/apache/storm/pull/700.diff","patch_url":"https://github.com/apache/storm/pull/700.patch"},"body":"This new feature is aimed for automatic flow control through the topology DAG since different components may have unmatched tuple processing speed. Currently, the tuples may get dropped if the downstream components can not process as quickly, thereby causing a waste of network bandwidth and processing capability. In addition, it is difficult to tune the max.spout.pending parameter for best backpressure performance. Therefore, an automatic back pressure scheme is highly desirable.\r\n\r\nRecently, Heron proposed a form of back pressure that does not rely on acking or max spout pending. Instead spouts throttle not only when max.spout.pending is hit, but also if any bolt has gone over a high water mark in their input queue, and has not yet gone below a low water mark again. There is a lot of room for potential improvement here around control theory and having spouts only respond to downstream bolts backing up, but a simple bang-bang controller like this is a great start.\r\n\r\nOur ABP scheme implements a light-weight yet efficient back pressure scheme. It monitors certain queues in executors and exploits the callback schemes on ZooKeeper and disruptor queue for a fast-responding (in a push manner) flow control.\r\n\r\nPlease check the attached figures for more details about the implementation.\r\nhttps://issues.apache.org/jira/secure/attachment/12752518/an%20simple%20example%20for%20backpressure.png\r\n![image](https://issues.apache.org/jira/secure/attachment/12752521/backpressure.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/storm/issues/comments/146779690","html_url":"https://github.com/apache/storm/pull/700#issuecomment-146779690","issue_url":"https://api.github.com/repos/apache/storm/issues/700","id":146779690,"user":{"login":"rsltrifork","id":1193502,"avatar_url":"https://avatars.githubusercontent.com/u/1193502?v=3","gravatar_id":"","url":"https://api.github.com/users/rsltrifork","html_url":"https://github.com/rsltrifork","followers_url":"https://api.github.com/users/rsltrifork/followers","following_url":"https://api.github.com/users/rsltrifork/following{/other_user}","gists_url":"https://api.github.com/users/rsltrifork/gists{/gist_id}","starred_url":"https://api.github.com/users/rsltrifork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsltrifork/subscriptions","organizations_url":"https://api.github.com/users/rsltrifork/orgs","repos_url":"https://api.github.com/users/rsltrifork/repos","events_url":"https://api.github.com/users/rsltrifork/events{/privacy}","received_events_url":"https://api.github.com/users/rsltrifork/received_events","type":"User","site_admin":false},"created_at":"2015-10-09T07:33:40Z","updated_at":"2015-10-09T07:33:40Z","body":"Thanks for the quick reply, revans2.\r\n\r\n> How do you know that the bolt is waiting in a controlled manner?\r\n\r\nA Bolt sending to an external system can include a circuit breaker, so it can keep waiting while the CB is tripped, while regularly retrying sends to check if the recipient is up.\r\n\r\nWhile doing this, the Bolt could also tell Storm that processing of the current tuple is currently blocked, and that Storm should reset the tuple timeout for it (and subsequent inflight tuples):\r\n            `collector.resetTupleTimeout(tuple);`\r\n\r\n> Even if you do know that how do you know that if it is waiting in a controlled manner that it has not lost track of a tuple?\r\n\r\nThe bolt is continuously saying that it still has the tuple."}}