{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/2606","labels_url":"https://api.github.com/repos/celery/celery/issues/2606/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/2606/comments","events_url":"https://api.github.com/repos/celery/celery/issues/2606/events","html_url":"https://github.com/celery/celery/issues/2606","id":73366175,"number":2606,"title":"worker getting stuck","user":{"login":"girishbin","id":1101219,"avatar_url":"https://avatars.githubusercontent.com/u/1101219?v=3","gravatar_id":"","url":"https://api.github.com/users/girishbin","html_url":"https://github.com/girishbin","followers_url":"https://api.github.com/users/girishbin/followers","following_url":"https://api.github.com/users/girishbin/following{/other_user}","gists_url":"https://api.github.com/users/girishbin/gists{/gist_id}","starred_url":"https://api.github.com/users/girishbin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/girishbin/subscriptions","organizations_url":"https://api.github.com/users/girishbin/orgs","repos_url":"https://api.github.com/users/girishbin/repos","events_url":"https://api.github.com/users/girishbin/events{/privacy}","received_events_url":"https://api.github.com/users/girishbin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-05T16:46:23Z","updated_at":"2015-05-05T18:27:53Z","closed_at":null,"body":"\r\nCelery worker getting stuck consuming lot of resident memory. \r\n\r\nVersion is  celery (3.1.17)\r\n\r\nStrace \r\n- - - - - - - - - - - - - - - - - - - - - - - - - -\r\n\r\ncelery]# strace  -p 8401\r\nProcess 8401 attached - interrupt to quit\r\nread(10,\r\n\r\n- - - - - - - - - - - - - - - - - - - - - - - - - -\r\n\r\ncelery]# lsof -n -p 8401 |   egrep -v '(DIR|REG)'\r\nCOMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME\r\npython  8401  dsl    0r  FIFO    0,8      0t0 124716100 pipe\r\npython  8401  dsl    1w  FIFO    0,8      0t0 124716101 pipe\r\npython  8401  dsl    2w  FIFO    0,8      0t0 124716101 pipe\r\npython  8401  dsl    6r  FIFO    0,8      0t0 124716462 pipe\r\npython  8401  dsl    7w  FIFO    0,8      0t0 124716462 pipe\r\npython  8401  dsl    8r  FIFO    0,8      0t0 124716463 pipe\r\npython  8401  dsl    9w  FIFO    0,8      0t0 124716463 pipe\r\npython  8401  dsl   10r  FIFO    0,8      0t0 124716464 pipe\r\npython  8401  dsl   13w  FIFO    0,8      0t0 124716465 pipe\r\npython  8401  dsl   14r  FIFO    0,8      0t0 124716466 pipe\r\npython  8401  dsl   15r   CHR    1,3      0t0      3662 /dev/null\r\npython  8401  dsl   16w  FIFO    0,8      0t0 124716467 pipe\r\n\r\n\r\nPstack dump\r\n- - - - - - - - - - - - - - - - - - - - - - - - - - \r\n\r\ncelery]# pstack 8401\r\n#0  0x0000003056c0e740 in __read_nocancel () from /lib64/libpthread.so.0\r\n#1  0x00007fa96b97b4c6 in _Billiard_conn_recvall () from /home/apps/analy/app/venv/lib/python2.6/site-packages/_billiard.so\r\n#2  0x00007fa96b97b552 in Billiard_conn_recv_string () from /home/apps/analy/app/venv/lib/python2.6/site-packages/_billiard.so\r\n#3  0x00007fa96b97b668 in Billiard_connection_recv_payload () from home/apps/analy/app/venv/lib/python2.6/site-packages/_billiard.so\r\n#4  0x00000030574d5916 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#5  0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#6  0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#7  0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#8  0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#9  0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#10 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#11 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#12 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#13 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#14 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#15 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#16 0x000000305746acb0 in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#17 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#18 0x00000030574566af in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#19 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#20 0x000000305749568e in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#21 0x0000003057494298 in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#22 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#23 0x00000030574d4f74 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#24 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#25 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#26 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#27 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#28 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#29 0x000000305746adad in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#30 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#31 0x00000030574d4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#32 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#33 0x000000305746adad in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#34 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#35 0x00000030574566af in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#36 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#37 0x000000305749568e in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#38 0x0000003057494298 in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#39 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#40 0x00000030574d4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#41 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#42 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#43 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#44 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#45 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#46 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#47 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#48 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#49 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#50 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#51 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#52 0x000000305746adad in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#53 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#54 0x00000030574d4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#55 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#56 0x000000305746adad in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#57 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#58 0x00000030574566af in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#59 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#60 0x0000003057495a54 in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#61 0x0000003057443c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0\r\n#62 0x00000030574d4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#63 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#64 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#65 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#66 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#67 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#68 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#69 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#70 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#71 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#72 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#73 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#74 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#75 0x00000030574d6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#76 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#77 0x00000030574d5aa4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0\r\n#78 0x00000030574d7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0\r\n#79 0x00000030574d7732 in PyEval_EvalCode () from /usr/lib64/libpython2.6.so.1.0\r\n#80 0x00000030574f1bac in ?? () from /usr/lib64/libpython2.6.so.1.0\r\n#81 0x00000030574f1c80 in PyRun_FileExFlags () from /usr/lib64/libpython2.6.so.1.0\r\n#82 0x00000030574f316c in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.6.so.1.0\r\n#83 0x00000030574ff8a2 in Py_Main () from /usr/lib64/libpython2.6.so.1.0\r\n#84 0x000000305681ed5d in __libc_start_main () from /lib64/libc.so.6\r\n#85 0x0000000000400649 in _start ()"},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/99168302","html_url":"https://github.com/celery/celery/issues/2606#issuecomment-99168302","issue_url":"https://api.github.com/repos/celery/celery/issues/2606","id":99168302,"user":{"login":"girishbin","id":1101219,"avatar_url":"https://avatars.githubusercontent.com/u/1101219?v=3","gravatar_id":"","url":"https://api.github.com/users/girishbin","html_url":"https://github.com/girishbin","followers_url":"https://api.github.com/users/girishbin/followers","following_url":"https://api.github.com/users/girishbin/following{/other_user}","gists_url":"https://api.github.com/users/girishbin/gists{/gist_id}","starred_url":"https://api.github.com/users/girishbin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/girishbin/subscriptions","organizations_url":"https://api.github.com/users/girishbin/orgs","repos_url":"https://api.github.com/users/girishbin/repos","events_url":"https://api.github.com/users/girishbin/events{/privacy}","received_events_url":"https://api.github.com/users/girishbin/received_events","type":"User","site_admin":false},"created_at":"2015-05-05T18:27:52Z","updated_at":"2015-05-05T18:27:52Z","body":"Its stuck on pipe read.\r\n\r\nproc]# ls -l /proc/8401/fd\r\ntotal 0\r\nlr-x------ 1 dsl dsl 64 May  5 17:26 0 -> pipe:[124716100]\r\nl-wx------ 1 dsl dsl 64 May  5 17:26 1 -> pipe:[124716101]\r\nlr-x------ 1 dsl dsl 64 May  5 17:26 10 -> pipe:[124716464]\r\nl-wx------ 1 dsl dsl 64 May  5 17:26 13 -> pipe:[124716465]\r\n"}}