{"action":"created","issue":{"url":"https://api.github.com/repos/intel-hadoop/gearpump/issues/1151","labels_url":"https://api.github.com/repos/intel-hadoop/gearpump/issues/1151/labels{/name}","comments_url":"https://api.github.com/repos/intel-hadoop/gearpump/issues/1151/comments","events_url":"https://api.github.com/repos/intel-hadoop/gearpump/issues/1151/events","html_url":"https://github.com/intel-hadoop/gearpump/issues/1151","id":93787269,"number":1151,"title":"scale up to 10, 000 tasks on single four-core machine","user":{"login":"clockfly","id":2595532,"avatar_url":"https://avatars.githubusercontent.com/u/2595532?v=3","gravatar_id":"","url":"https://api.github.com/users/clockfly","html_url":"https://github.com/clockfly","followers_url":"https://api.github.com/users/clockfly/followers","following_url":"https://api.github.com/users/clockfly/following{/other_user}","gists_url":"https://api.github.com/users/clockfly/gists{/gist_id}","starred_url":"https://api.github.com/users/clockfly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clockfly/subscriptions","organizations_url":"https://api.github.com/users/clockfly/orgs","repos_url":"https://api.github.com/users/clockfly/repos","events_url":"https://api.github.com/users/clockfly/events{/privacy}","received_events_url":"https://api.github.com/users/clockfly/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/intel-hadoop/gearpump/labels/feature","name":"feature","color":"0052cc"},{"url":"https://api.github.com/repos/intel-hadoop/gearpump/labels/P2","name":"P2","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-07-08T13:18:08Z","updated_at":"2015-07-14T13:58:44Z","closed_at":null,"body":"In theory, with effecitve flow control, we should be able to scale to as many tasks as we want.\r\n\r\nBut in reality, it depends on:\r\n1. memory footprint of single task is small enough, so that there is no out of memory.\r\n2. GC pause interval is still under control to avoid remote watching timeout."},"comment":{"url":"https://api.github.com/repos/intel-hadoop/gearpump/issues/comments/121246648","html_url":"https://github.com/intel-hadoop/gearpump/issues/1151#issuecomment-121246648","issue_url":"https://api.github.com/repos/intel-hadoop/gearpump/issues/1151","id":121246648,"user":{"login":"clockfly","id":2595532,"avatar_url":"https://avatars.githubusercontent.com/u/2595532?v=3","gravatar_id":"","url":"https://api.github.com/users/clockfly","html_url":"https://github.com/clockfly","followers_url":"https://api.github.com/users/clockfly/followers","following_url":"https://api.github.com/users/clockfly/following{/other_user}","gists_url":"https://api.github.com/users/clockfly/gists{/gist_id}","starred_url":"https://api.github.com/users/clockfly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clockfly/subscriptions","organizations_url":"https://api.github.com/users/clockfly/orgs","repos_url":"https://api.github.com/users/clockfly/repos","events_url":"https://api.github.com/users/clockfly/events{/privacy}","received_events_url":"https://api.github.com/users/clockfly/received_events","type":"User","site_admin":false},"created_at":"2015-07-14T13:58:44Z","updated_at":"2015-07-14T13:58:44Z","body":"For a 2GB memory machine, we can at most support 8000 task shuffle to another 8000 tasks.\r\n\r\nIn these 35 bytes. \r\n1. 13 bytes used to store the received message count. (In theory, we only need 6 bytes, with 4 bytes as key, and 2 bytes as value. But as it is a hashmap, 2x the space will be used.)\r\n2. 8 bytes to store the min-clock for current sliding window.\r\n3. 8 bytes to store the candidate min-clock for current sliding window.\r\n4. 2 byte to store sent message count.\r\n5. 2 byte to store pending message count.\r\n6. 2 bytes to store threshold message count. (the sent message > threshold, then candidate clock is effective).\r\n\r\nThere maybe some room in 1, 2. 3. It may be possible to reduce 35 bytes to 14 bytes. Then in theory, for 2GB memory, we can support 12000 tasks shuffle to another 12000 tasks.\r\n\r\n\r\n "}}