{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/presto/issues/3390","labels_url":"https://api.github.com/repos/facebook/presto/issues/3390/labels{/name}","comments_url":"https://api.github.com/repos/facebook/presto/issues/3390/comments","events_url":"https://api.github.com/repos/facebook/presto/issues/3390/events","html_url":"https://github.com/facebook/presto/issues/3390","id":98493512,"number":3390,"title":"presto job hang when large data on csv hive table.","user":{"login":"tonywutao","id":672649,"avatar_url":"https://avatars.githubusercontent.com/u/672649?v=3","gravatar_id":"","url":"https://api.github.com/users/tonywutao","html_url":"https://github.com/tonywutao","followers_url":"https://api.github.com/users/tonywutao/followers","following_url":"https://api.github.com/users/tonywutao/following{/other_user}","gists_url":"https://api.github.com/users/tonywutao/gists{/gist_id}","starred_url":"https://api.github.com/users/tonywutao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonywutao/subscriptions","organizations_url":"https://api.github.com/users/tonywutao/orgs","repos_url":"https://api.github.com/users/tonywutao/repos","events_url":"https://api.github.com/users/tonywutao/events{/privacy}","received_events_url":"https://api.github.com/users/tonywutao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-01T00:41:06Z","updated_at":"2015-08-01T18:38:19Z","closed_at":null,"body":"hi, guys\r\n\r\nI am testing presto for PRD usage. I encountered a presto job hang problem.\r\n\r\nenvironment\r\n* Centos 6.5\r\n* Oracle jdk1.8_51\r\n* prest0 0.112\r\n* CDH5.4.0 (Hadoop-2.6.0-cdh, Hive-1.1-cdh)\r\n* 1 node as coordinator, and other 15 as worker. 16GB Heapsize for both coordinator and worker\r\n\r\nwe have a hive (CTRL A split  File) table 40+GB  per day with partition\r\n```shell\r\n[ttt@server ~]$ hadoop fs -du -s -h /user/ddd/traffic/page/dt=20150730\r\n47.0 G  141.0 G  /user/ddd/traffic/page/dt=20150730\r\n[ttt@server ~]$ hadoop fs -du -s -h /user/ddd/traffic/page/dt=20150729\r\n46.2 G  138.7 G  /user/ddd/traffic/page/dt=20150729\r\n```\r\ncase1: this sql on one day partition data works good. it costed 35 seconds.\r\n```shell\r\npresto:mars> select c_ip, cs_uri_stem, cs_host, count(cs_uri_stem) ct, cs_method, sc_status, count( distinct x_session_id) sessioncount from traffic_page where dt='20150730' and lower(cs_host) like 'secure%' and c_ip<>'' and c_ip <> '204.14.213.149' group by c_ip, cs_uri_stem,cs_host,cs_method,sc_status having count(cs_uri_stem)>500 order by ct desc limit 1000;\r\n\r\nQuery 20150731_222950_00004_uiau7, FINISHED, 16 nodes\r\nSplits: 305 total, 305 done (100.00%)\r\n0:35 [84.4M rows, 47GB] [2.41M rows/s, 1.35GB/s]\r\n```\r\n\r\ncase 2: this sql on 2-3 or more days data will hang. costed over 7 minutes, then i aborted it\r\n```shell\r\npresto:mars> select c_ip, cs_uri_stem, cs_host, count(cs_uri_stem) ct, cs_method, sc_status, count( distinct x_session_id) sessioncount from traffic_page where ( dt='20150730' or dt='20150729' ) and lower(cs_host) like 'secure%' and c_ip<>'' and c_ip <> '204.14.213.149' group by c_ip, cs_uri_stem,cs_host,cs_method,sc_status having count(cs_uri_stem)>500 order by ct desc limit 1000;\r\n\r\nQuery 20150731_223141_00006_uiau7, RUNNING, 16 nodes\r\nSplits: 593 total, 576 done (97.13%)\r\n7:31 [170M rows, 93.2GB] [378K rows/s, 212MB/s]\r\n\r\nQuery aborted by user\r\n```\r\n\r\nI opened the Jmx to watch the system/jvm metrics, i found when the presto job hang, cpu/network is really low, memory is no problem. i just saw the splitRunner are waiting when job hang. \r\n![work-bad](https://cloud.githubusercontent.com/assets/672649/9019964/4fb12b00-37ab-11e5-9f98-aada03d3f109.PNG)\r\n![work-good](https://cloud.githubusercontent.com/assets/672649/9019963/4fafbfcc-37ab-11e5-8c4a-bb0bf296efe3.PNG)\r\n\r\n\r\nso when the data size are too large, the splitRunner will hang?\r\n\r\nany suggestion? Thanks in advance\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/facebook/presto/issues/comments/126944091","html_url":"https://github.com/facebook/presto/issues/3390#issuecomment-126944091","issue_url":"https://api.github.com/repos/facebook/presto/issues/3390","id":126944091,"user":{"login":"tonywutao","id":672649,"avatar_url":"https://avatars.githubusercontent.com/u/672649?v=3","gravatar_id":"","url":"https://api.github.com/users/tonywutao","html_url":"https://github.com/tonywutao","followers_url":"https://api.github.com/users/tonywutao/followers","following_url":"https://api.github.com/users/tonywutao/following{/other_user}","gists_url":"https://api.github.com/users/tonywutao/gists{/gist_id}","starred_url":"https://api.github.com/users/tonywutao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonywutao/subscriptions","organizations_url":"https://api.github.com/users/tonywutao/orgs","repos_url":"https://api.github.com/users/tonywutao/repos","events_url":"https://api.github.com/users/tonywutao/events{/privacy}","received_events_url":"https://api.github.com/users/tonywutao/received_events","type":"User","site_admin":false},"created_at":"2015-08-01T18:38:19Z","updated_at":"2015-08-01T18:38:19Z","body":"fixed by this setting for worker\r\ntask.max-memory=4GB\r\n"}}