{"action":"created","issue":{"url":"https://api.github.com/repos/rundeck/rundeck/issues/1140","labels_url":"https://api.github.com/repos/rundeck/rundeck/issues/1140/labels{/name}","comments_url":"https://api.github.com/repos/rundeck/rundeck/issues/1140/comments","events_url":"https://api.github.com/repos/rundeck/rundeck/issues/1140/events","html_url":"https://github.com/rundeck/rundeck/issues/1140","id":61215476,"number":1140,"title":"Jobs getting 'stuck' after 2.4.0 upgrade","user":{"login":"schicky","id":7059792,"avatar_url":"https://avatars.githubusercontent.com/u/7059792?v=3","gravatar_id":"","url":"https://api.github.com/users/schicky","html_url":"https://github.com/schicky","followers_url":"https://api.github.com/users/schicky/followers","following_url":"https://api.github.com/users/schicky/following{/other_user}","gists_url":"https://api.github.com/users/schicky/gists{/gist_id}","starred_url":"https://api.github.com/users/schicky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schicky/subscriptions","organizations_url":"https://api.github.com/users/schicky/orgs","repos_url":"https://api.github.com/users/schicky/repos","events_url":"https://api.github.com/users/schicky/events{/privacy}","received_events_url":"https://api.github.com/users/schicky/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-13T20:10:45Z","updated_at":"2015-03-17T14:01:08Z","closed_at":"2015-03-13T21:29:51Z","body":"After upgrading to 2.4.0 we've had jobs get 'stuck' where there is nothing running on any of the nodes but the job keep showing running. The screen shot shows that, and if you look at the log output there is nothing new. I check the connections to the servers and don't see anything open. Usually the jobs finish up eventually. For example we have a job run for 30 minutes but it finished the last node in 15 minutes so it does 15 minutes of nothing. We recently we had a job run on 8 nodes and finish in a minute but kept running for almost 3 days until we killed it.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/7059792/6646482/8bd77ce8-c99b-11e4-990b-c12cae03560d.png)\r\n"},"comment":{"url":"https://api.github.com/repos/rundeck/rundeck/issues/comments/82368084","html_url":"https://github.com/rundeck/rundeck/issues/1140#issuecomment-82368084","issue_url":"https://api.github.com/repos/rundeck/rundeck/issues/1140","id":82368084,"user":{"login":"schicky","id":7059792,"avatar_url":"https://avatars.githubusercontent.com/u/7059792?v=3","gravatar_id":"","url":"https://api.github.com/users/schicky","html_url":"https://github.com/schicky","followers_url":"https://api.github.com/users/schicky/followers","following_url":"https://api.github.com/users/schicky/following{/other_user}","gists_url":"https://api.github.com/users/schicky/gists{/gist_id}","starred_url":"https://api.github.com/users/schicky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schicky/subscriptions","organizations_url":"https://api.github.com/users/schicky/orgs","repos_url":"https://api.github.com/users/schicky/repos","events_url":"https://api.github.com/users/schicky/events{/privacy}","received_events_url":"https://api.github.com/users/schicky/received_events","type":"User","site_admin":false},"created_at":"2015-03-17T14:01:08Z","updated_at":"2015-03-17T14:01:08Z","body":"I'm using tomcat so I assume the equivalent is catalina.out and I didnt see anything in there. I turned on debug logging and saw some strange things. You can see the gap in time, this is a smaller gap as usually and it seems to increase when the number of nodes I execute on increases. It also lists node failures but all the nodes on the job are reported as successful. \r\n\r\n![image](https://cloud.githubusercontent.com/assets/7059792/6688738/2e646a3c-cc8c-11e4-8fb9-d404ad7a8378.png)\r\n"}}