{"action":"created","issue":{"url":"https://api.github.com/repos/clustervision/trinity/issues/506","labels_url":"https://api.github.com/repos/clustervision/trinity/issues/506/labels{/name}","comments_url":"https://api.github.com/repos/clustervision/trinity/issues/506/comments","events_url":"https://api.github.com/repos/clustervision/trinity/issues/506/events","html_url":"https://github.com/clustervision/trinity/issues/506","id":116101022,"number":506,"title":"slurm reports containers are down (after a time lapse) [ they are not]","user":{"login":"themisa","id":6388639,"avatar_url":"https://avatars.githubusercontent.com/u/6388639?v=3","gravatar_id":"","url":"https://api.github.com/users/themisa","html_url":"https://github.com/themisa","followers_url":"https://api.github.com/users/themisa/followers","following_url":"https://api.github.com/users/themisa/following{/other_user}","gists_url":"https://api.github.com/users/themisa/gists{/gist_id}","starred_url":"https://api.github.com/users/themisa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/themisa/subscriptions","organizations_url":"https://api.github.com/users/themisa/orgs","repos_url":"https://api.github.com/users/themisa/repos","events_url":"https://api.github.com/users/themisa/events{/privacy}","received_events_url":"https://api.github.com/users/themisa/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/clustervision/trinity/labels/ASAP","name":"ASAP","color":"fbca04"},{"url":"https://api.github.com/repos/clustervision/trinity/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/clustervision/trinity/labels/r6","name":"r6","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"themisa","id":6388639,"avatar_url":"https://avatars.githubusercontent.com/u/6388639?v=3","gravatar_id":"","url":"https://api.github.com/users/themisa","html_url":"https://github.com/themisa","followers_url":"https://api.github.com/users/themisa/followers","following_url":"https://api.github.com/users/themisa/following{/other_user}","gists_url":"https://api.github.com/users/themisa/gists{/gist_id}","starred_url":"https://api.github.com/users/themisa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/themisa/subscriptions","organizations_url":"https://api.github.com/users/themisa/orgs","repos_url":"https://api.github.com/users/themisa/repos","events_url":"https://api.github.com/users/themisa/events{/privacy}","received_events_url":"https://api.github.com/users/themisa/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/clustervision/trinity/milestones/7","html_url":"https://github.com/clustervision/trinity/milestones/M7:%20Grumpy%20Gargoyle","labels_url":"https://api.github.com/repos/clustervision/trinity/milestones/7/labels","id":1303835,"number":7,"title":"M7: Grumpy Gargoyle","description":"","creator":{"login":"hansthen","id":2202342,"avatar_url":"https://avatars.githubusercontent.com/u/2202342?v=3","gravatar_id":"","url":"https://api.github.com/users/hansthen","html_url":"https://github.com/hansthen","followers_url":"https://api.github.com/users/hansthen/followers","following_url":"https://api.github.com/users/hansthen/following{/other_user}","gists_url":"https://api.github.com/users/hansthen/gists{/gist_id}","starred_url":"https://api.github.com/users/hansthen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hansthen/subscriptions","organizations_url":"https://api.github.com/users/hansthen/orgs","repos_url":"https://api.github.com/users/hansthen/repos","events_url":"https://api.github.com/users/hansthen/events{/privacy}","received_events_url":"https://api.github.com/users/hansthen/received_events","type":"User","site_admin":false},"open_issues":33,"closed_issues":23,"state":"open","created_at":"2015-09-14T08:16:26Z","updated_at":"2015-11-11T22:19:20Z","due_on":"2015-12-06T23:00:00Z","closed_at":null},"comments":4,"created_at":"2015-11-10T12:53:30Z","updated_at":"2015-11-12T16:29:35Z","closed_at":null,"body":"DISCLAIMER: this issue needs to be more closely monitored.\r\n\r\n**Version:** r6 4eb9450\r\n**Environment:** amaster\r\n\r\nFrom login.vc-a: sinfo reports correct containers. After some time lapse (~15-20 min), containers reported down. If i do system trinity force-reload, containers reported as up.\r\n\r\nAgain, if i leave system for a while, containers reported as down again from  sinfo.\r\n\r\n\r\nThis is the output from the slurm log file:\r\n```\r\n[2015-11-10T12:11:19.624] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=4,partition_job_depth=0\r\n[2015-11-10T12:15:57.027] node c002 returned to service\r\n[2015-11-10T12:15:58.232] node c001 returned to service\r\n[2015-11-10T12:20:19.846] error: Nodes c002 not responding\r\n[2015-11-10T12:20:22.953] Node c002 now responding\r\n[2015-11-10T12:25:19.010] error: Nodes c002 not responding\r\n[2015-11-10T12:25:22.123] Node c002 now responding\r\n[2015-11-10T12:30:19.173] error: Nodes c001 not responding\r\n[2015-11-10T12:32:02.284] error: Nodes c001 not responding, setting DOWN\r\n[2015-11-10T12:38:42.447] error: Nodes c002 not responding, setting DOWN\r\n```\r\n\r\nIf we login to compute nodes though, the containers are up and running:\r\n```\r\n[root@controller(amaster*) ~]# ssh node001\r\nLast login: Tue Nov 10 12:17:42 2015 from controller.cluster\r\n[root@node001 ~]# docker ps\r\nCONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS               NAMES\r\n5f0dce2d0ad7        controller:5050/trinity:latest   \"/usr/bin/supervisor   About an hour ago   Up About an hour                        trinity        \r\n```\r\n\r\n```\r\n[root@node001 ~]# systemctl status trinity\r\ntrinity.service - LSB: start and stop trinity\r\n   Loaded: loaded (/etc/rc.d/init.d/trinity)\r\n   Active: active (exited) since Tue 2015-11-10 11:13:01 CET; 2h 39min ago\r\n  Process: 3758 ExecStop=/etc/rc.d/init.d/trinity stop (code=exited, status=0/SUCCESS)\r\n  Process: 4558 ExecReload=/etc/rc.d/init.d/trinity reload (code=exited, status=0/SUCCESS)\r\n  Process: 3766 ExecStart=/etc/rc.d/init.d/trinity start (code=exited, status=0/SUCCESS)\r\n\r\nNov 10 12:15:48 node001.cluster trinity[4558]: PING login-a.vc-a (172.16.255.254) 56(84) bytes of data.\r\nNov 10 12:15:48 node001.cluster trinity[4558]: 64 bytes from login-a.vc-a (172.16.255.254): icmp_seq=1 ttl=63 time=0.467 ms\r\nNov 10 12:15:49 node001.cluster trinity[4558]: 64 bytes from login-a.vc-a (172.16.255.254): icmp_seq=2 ttl=63 time=0.633 ms\r\nNov 10 12:15:50 node001.cluster trinity[4558]: 64 bytes from login-a.vc-a (172.16.255.254): icmp_seq=3 ttl=63 time=0.532 ms\r\nNov 10 12:15:51 node001.cluster trinity[4558]: 64 bytes from login-a.vc-a (172.16.255.254): icmp_seq=4 ttl=63 time=0.496 ms\r\nNov 10 12:15:52 node001.cluster trinity[4558]: 64 bytes from login-a.vc-a (172.16.255.254): icmp_seq=5 ttl=63 time=0.601 ms\r\nNov 10 12:15:52 node001.cluster trinity[4558]: --- login-a.vc-a ping statistics ---\r\nNov 10 12:15:52 node001.cluster trinity[4558]: 5 packets transmitted, 5 received, 0% packet loss, time 4005ms\r\nNov 10 12:15:52 node001.cluster trinity[4558]: rtt min/avg/max/mdev = 0.467/0.545/0.633/0.069 ms\r\nNov 10 12:15:56 node001.cluster trinity[4558]: 5f0dce2d0ad7981a8e8e2ba661c47c8998995f86acce2e430e8fe481503e7ffe\r\n```"},"comment":{"url":"https://api.github.com/repos/clustervision/trinity/issues/comments/156156476","html_url":"https://github.com/clustervision/trinity/issues/506#issuecomment-156156476","issue_url":"https://api.github.com/repos/clustervision/trinity/issues/506","id":156156476,"user":{"login":"themisa","id":6388639,"avatar_url":"https://avatars.githubusercontent.com/u/6388639?v=3","gravatar_id":"","url":"https://api.github.com/users/themisa","html_url":"https://github.com/themisa","followers_url":"https://api.github.com/users/themisa/followers","following_url":"https://api.github.com/users/themisa/following{/other_user}","gists_url":"https://api.github.com/users/themisa/gists{/gist_id}","starred_url":"https://api.github.com/users/themisa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/themisa/subscriptions","organizations_url":"https://api.github.com/users/themisa/orgs","repos_url":"https://api.github.com/users/themisa/repos","events_url":"https://api.github.com/users/themisa/events{/privacy}","received_events_url":"https://api.github.com/users/themisa/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T16:29:35Z","updated_at":"2015-11-12T16:29:35Z","body":"**Version:** r6 cc03c73\r\n**Environment:** amaster\r\n\r\nNot sure this is fixed.\r\n\r\n```\r\n[root@login ~]# cat /var/log/slurm/slurmctld.log \r\n\r\n[2015-11-12T17:23:30.015] error: authentication: Invalid credential \r\n[2015-11-12T17:23:30.015] error: slurm_receive_msg: Protocol authentication error\r\n[2015-11-12T17:23:30.026] error: slurm_receive_msg: Protocol authentication error\r\n[2015-11-12T17:23:30.306] error: Munge decode failed: Invalid credential\r\n[2015-11-12T17:23:30.306] error: authentication: Invalid credential \r\n[2015-11-12T17:23:30.306] error: slurm_receive_msg: Protocol authentication error\r\n[2015-11-12T17:23:30.316] error: slurm_receive_msg: Protocol authentication error\r\n[root@login ~]# sinfo\r\nPARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST\r\ncontainer*    up   infinite      2   unk* c[001-002]\r\n```"}}