{"action":"opened","issue":{"url":"https://api.github.com/repos/clustervision/trinity/issues/424","labels_url":"https://api.github.com/repos/clustervision/trinity/issues/424/labels{/name}","comments_url":"https://api.github.com/repos/clustervision/trinity/issues/424/comments","events_url":"https://api.github.com/repos/clustervision/trinity/issues/424/events","html_url":"https://github.com/clustervision/trinity/issues/424","id":106298763,"number":424,"title":"Munge error in the containers","user":{"login":"abhimukh","id":8803654,"avatar_url":"https://avatars.githubusercontent.com/u/8803654?v=3","gravatar_id":"","url":"https://api.github.com/users/abhimukh","html_url":"https://github.com/abhimukh","followers_url":"https://api.github.com/users/abhimukh/followers","following_url":"https://api.github.com/users/abhimukh/following{/other_user}","gists_url":"https://api.github.com/users/abhimukh/gists{/gist_id}","starred_url":"https://api.github.com/users/abhimukh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhimukh/subscriptions","organizations_url":"https://api.github.com/users/abhimukh/orgs","repos_url":"https://api.github.com/users/abhimukh/repos","events_url":"https://api.github.com/users/abhimukh/events{/privacy}","received_events_url":"https://api.github.com/users/abhimukh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-14T08:56:23Z","updated_at":"2015-09-14T08:56:23Z","closed_at":null,"body":"```\r\n[root@trinity(demo) ~]# cat /trinity/version\r\n* r5\r\n34a01f3\r\n```\r\n(demo env)\r\n\r\n```\r\n[root@login ~]# sinfo\r\nPARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST\r\ncontainer*    up   infinite      2   down c[001-002]\r\ncontainer*    up   infinite      1    unk c004\r\n\r\n```\r\n```\r\n[root@c001 /]# sinfo                         \r\nsinfo: error: slurm_receive_msg: Zero Bytes were transmitted or received\r\nslurm_load_partitions: Zero Bytes were transmitted or received\r\n[root@c001 /]# cat /var/log/slurm/slurmd.log \r\n\r\n//...\r\n2015-09-14T06:44:37.084] error: Munge decode failed: Expired credential\r\n[2015-09-14T06:44:37.084] ENCODED: Mon Sep 14 06:39:33 2015\r\n[2015-09-14T06:44:37.084] DECODED: Mon Sep 14 06:44:37 2015\r\n//...\r\n```\r\nThis seems to be related to clock synchronization\r\nhttps://computing.llnl.gov/linux/slurm/faq.html#clock\r\n\r\n```\r\n[root@login ~]# date\r\nma sep 14 08:55:20 UTC 2015\r\n```\r\n```\r\n[root@c001 /]# date                          \r\nMon Sep 14 09:00:26 UTC 2015\r\n```\r\n\r\nThere is a big drift between the login node and the containers"}}