{"action":"created","issue":{"url":"https://api.github.com/repos/pyfarm/pyfarm-agent/issues/139","labels_url":"https://api.github.com/repos/pyfarm/pyfarm-agent/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/pyfarm/pyfarm-agent/issues/139/comments","events_url":"https://api.github.com/repos/pyfarm/pyfarm-agent/issues/139/events","html_url":"https://github.com/pyfarm/pyfarm-agent/issues/139","id":41719870,"number":139,"title":"Agent gets in endless loop updating state of tasks","user":{"login":"guidow","id":6033185,"avatar_url":"https://avatars.githubusercontent.com/u/6033185?v=3","gravatar_id":"","url":"https://api.github.com/users/guidow","html_url":"https://github.com/guidow","followers_url":"https://api.github.com/users/guidow/followers","following_url":"https://api.github.com/users/guidow/following{/other_user}","gists_url":"https://api.github.com/users/guidow/gists{/gist_id}","starred_url":"https://api.github.com/users/guidow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guidow/subscriptions","organizations_url":"https://api.github.com/users/guidow/orgs","repos_url":"https://api.github.com/users/guidow/repos","events_url":"https://api.github.com/users/guidow/events{/privacy}","received_events_url":"https://api.github.com/users/guidow/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pyfarm/pyfarm-agent/labels/agent","name":"agent","color":"009800"},{"url":"https://api.github.com/repos/pyfarm/pyfarm-agent/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"guidow","id":6033185,"avatar_url":"https://avatars.githubusercontent.com/u/6033185?v=3","gravatar_id":"","url":"https://api.github.com/users/guidow","html_url":"https://github.com/guidow","followers_url":"https://api.github.com/users/guidow/followers","following_url":"https://api.github.com/users/guidow/following{/other_user}","gists_url":"https://api.github.com/users/guidow/gists{/gist_id}","starred_url":"https://api.github.com/users/guidow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guidow/subscriptions","organizations_url":"https://api.github.com/users/guidow/orgs","repos_url":"https://api.github.com/users/guidow/repos","events_url":"https://api.github.com/users/guidow/events{/privacy}","received_events_url":"https://api.github.com/users/guidow/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2014-09-02T15:11:56Z","updated_at":"2015-06-03T15:29:52Z","closed_at":"2015-06-03T15:29:52Z","body":"I just found the agent in an endless loop trying to set all tasks in an assignment to failed like this:\r\n\r\n    {\r\n        \"time_finished\": \"2014-09-02T14:34:20.053165\", \r\n        \"time_started\": \"2014-09-02T14:33:28.272774\", \r\n        \"time_submitted\": \"2014-09-02T14:27:12.390892\", \r\n        \"last_error\": \"Traceback (most recent call last):\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\twisted\\\\internet\\\\base.py\\\", line 1201, in mainLoop\\n    self.runUntilCurrent()\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\twisted\\\\internet\\\\base.py\\\", line 797, in runUntilCurrent\\n    f(*a, **kw)\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\twisted\\\\internet\\\\defer.py\\\", line 383, in callback\\n    self._startRunCallbacks(result)\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\twisted\\\\internet\\\\defer.py\\\", line 491, in _startRunCallbacks\\n    self._runCallbacks()\\n--- <exception caught here> ---\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\twisted\\\\internet\\\\defer.py\\\", line 578, in _runCallbacks\\n    current.result = callback(current.result, *args, **kw)\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\agent\\\\http\\\\api\\\\assign.py\\\", line 266, in loaded_jobtype\\n    started_deferred, stopped_deferred = instance._start()\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\jobtypes\\\\core\\\\internals.py\\\", line 340, in _start\\n    self.start()\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\jobtypes\\\\core\\\\jobtype.py\\\", line 686, in start\\n    self._spawn_process(command)\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\jobtypes\\\\core\\\\jobtype.py\\\", line 612, in _spawn_process\\n    self._register_logfile_on_master(basename(log_path))\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\jobtypes\\\\core\\\\jobtype.py\\\", line 665, in _register_logfile_on_master\\n    post_logfile(task, log_path)\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\jobtypes\\\\core\\\\jobtype.py\\\", line 622, in post_logfile\\n    \\\"agent_id\\\": self.node()[\\\"id\\\"]}\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\jobtypes\\\\core\\\\jobtype.py\\\", line 309, in node\\n    \\\"consumed_ram\\\": int(memory.total_consumption()),\\n  File \\\"c:\\\\svn\\\\PyFarm\\\\src\\\\pyfarm-agent\\\\pyfarm\\\\agent\\\\sysinfo\\\\memory.py\\\", line 60, in total_consumption\\n    total += child_process.memory_info().rss\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\psutil\\\\__init__.py\\\", line 889, in memory_info\\n    return self._proc.memory_info()\\n  File \\\"C:\\\\Ambient\\\\PythonPyFarm\\\\lib\\\\site-packages\\\\psutil\\\\_pswindows.py\\\", line 215, in wrapper\\n    raise NoSuchProcess(self.pid, self._name)\\npsutil.NoSuchProcess: process no longer exists (pid=3672)\\n\", \r\n        \"frame\": 104.0, \r\n        \"state\": \"failed\", \r\n        \"priority\": 0, \r\n        \"attempts\": 4, \r\n        \"agent_id\": 1, \r\n        \"hidden\": false, \r\n        \"project_id\": null, \r\n        \"id\": 2093, \r\n        \"job_id\": 42\r\n      }\r\n\r\nI still haven't figured out what caused the endless loop, this should be investigated.\r\n\r\nIn the meantime, memory.total_consumption() should catch NoSuchProcess to deal with the apparent race condition."},"comment":{"url":"https://api.github.com/repos/pyfarm/pyfarm-agent/issues/comments/108485099","html_url":"https://github.com/pyfarm/pyfarm-agent/issues/139#issuecomment-108485099","issue_url":"https://api.github.com/repos/pyfarm/pyfarm-agent/issues/139","id":108485099,"user":{"login":"guidow","id":6033185,"avatar_url":"https://avatars.githubusercontent.com/u/6033185?v=3","gravatar_id":"","url":"https://api.github.com/users/guidow","html_url":"https://github.com/guidow","followers_url":"https://api.github.com/users/guidow/followers","following_url":"https://api.github.com/users/guidow/following{/other_user}","gists_url":"https://api.github.com/users/guidow/gists{/gist_id}","starred_url":"https://api.github.com/users/guidow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guidow/subscriptions","organizations_url":"https://api.github.com/users/guidow/orgs","repos_url":"https://api.github.com/users/guidow/repos","events_url":"https://api.github.com/users/guidow/events{/privacy}","received_events_url":"https://api.github.com/users/guidow/received_events","type":"User","site_admin":false},"created_at":"2015-06-03T15:29:52Z","updated_at":"2015-06-03T15:29:52Z","body":"This problem hasn't resurfaced in a long time, assuming it is fixed by now."}}