{"action":"created","issue":{"url":"https://api.github.com/repos/clustervision/trinity/issues/67","labels_url":"https://api.github.com/repos/clustervision/trinity/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/clustervision/trinity/issues/67/comments","events_url":"https://api.github.com/repos/clustervision/trinity/issues/67/events","html_url":"https://github.com/clustervision/trinity/issues/67","id":58188379,"number":67,"title":"configured nodes become unreachable every morning","user":{"login":"themisa","id":6388639,"avatar_url":"https://avatars.githubusercontent.com/u/6388639?v=3","gravatar_id":"","url":"https://api.github.com/users/themisa","html_url":"https://github.com/themisa","followers_url":"https://api.github.com/users/themisa/followers","following_url":"https://api.github.com/users/themisa/following{/other_user}","gists_url":"https://api.github.com/users/themisa/gists{/gist_id}","starred_url":"https://api.github.com/users/themisa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/themisa/subscriptions","organizations_url":"https://api.github.com/users/themisa/orgs","repos_url":"https://api.github.com/users/themisa/repos","events_url":"https://api.github.com/users/themisa/events{/privacy}","received_events_url":"https://api.github.com/users/themisa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"linuxstudio","id":7702698,"avatar_url":"https://avatars.githubusercontent.com/u/7702698?v=3","gravatar_id":"","url":"https://api.github.com/users/linuxstudio","html_url":"https://github.com/linuxstudio","followers_url":"https://api.github.com/users/linuxstudio/followers","following_url":"https://api.github.com/users/linuxstudio/following{/other_user}","gists_url":"https://api.github.com/users/linuxstudio/gists{/gist_id}","starred_url":"https://api.github.com/users/linuxstudio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linuxstudio/subscriptions","organizations_url":"https://api.github.com/users/linuxstudio/orgs","repos_url":"https://api.github.com/users/linuxstudio/repos","events_url":"https://api.github.com/users/linuxstudio/events{/privacy}","received_events_url":"https://api.github.com/users/linuxstudio/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/clustervision/trinity/milestones/2","html_url":"https://github.com/clustervision/trinity/milestones/M3","labels_url":"https://api.github.com/repos/clustervision/trinity/milestones/2/labels","id":938190,"number":2,"title":"M3","description":"","creator":{"login":"hansthen","id":2202342,"avatar_url":"https://avatars.githubusercontent.com/u/2202342?v=3","gravatar_id":"","url":"https://api.github.com/users/hansthen","html_url":"https://github.com/hansthen","followers_url":"https://api.github.com/users/hansthen/followers","following_url":"https://api.github.com/users/hansthen/following{/other_user}","gists_url":"https://api.github.com/users/hansthen/gists{/gist_id}","starred_url":"https://api.github.com/users/hansthen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hansthen/subscriptions","organizations_url":"https://api.github.com/users/hansthen/orgs","repos_url":"https://api.github.com/users/hansthen/repos","events_url":"https://api.github.com/users/hansthen/events{/privacy}","received_events_url":"https://api.github.com/users/hansthen/received_events","type":"User","site_admin":false},"open_issues":18,"closed_issues":25,"state":"open","created_at":"2015-01-16T12:17:59Z","updated_at":"2015-02-27T16:01:08Z","due_on":"2017-03-05T23:00:00Z","closed_at":null},"comments":21,"created_at":"2015-02-19T10:08:04Z","updated_at":"2015-03-03T10:04:21Z","closed_at":null,"body":"**Software:** Trinity r3\r\n**Component:** nodeset images\r\n**Hardware:** Supermicro, Dell PowerEdge\r\n**Test case:** general\r\n**Bug description:**\r\nWe have been consistently noticing that nodes that have been nodeset either with controller, openstack, compute, trinity images become unpingable the next day and need a power reset, and sometimes a second nodeset. We should keep track of these incidents, and check whether it is \"human error\" related (people powering down nodes at night) or if the system is unstable.\r\n"},"comment":{"url":"https://api.github.com/repos/clustervision/trinity/issues/comments/76918019","html_url":"https://github.com/clustervision/trinity/issues/67#issuecomment-76918019","issue_url":"https://api.github.com/repos/clustervision/trinity/issues/67","id":76918019,"user":{"login":"brunopereira81","id":5538691,"avatar_url":"https://avatars.githubusercontent.com/u/5538691?v=3","gravatar_id":"","url":"https://api.github.com/users/brunopereira81","html_url":"https://github.com/brunopereira81","followers_url":"https://api.github.com/users/brunopereira81/followers","following_url":"https://api.github.com/users/brunopereira81/following{/other_user}","gists_url":"https://api.github.com/users/brunopereira81/gists{/gist_id}","starred_url":"https://api.github.com/users/brunopereira81/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brunopereira81/subscriptions","organizations_url":"https://api.github.com/users/brunopereira81/orgs","repos_url":"https://api.github.com/users/brunopereira81/repos","events_url":"https://api.github.com/users/brunopereira81/events{/privacy}","received_events_url":"https://api.github.com/users/brunopereira81/received_events","type":"User","site_admin":false},"created_at":"2015-03-03T10:04:21Z","updated_at":"2015-03-03T10:04:21Z","body":"I think are seeing this wrong: its not that the nodes are not requesting a new IP address from the DHCP server, it is that the DHCP server never returns a valid address to the node and the node ends up stopping requesting the address and the whole services get scrambled.\r\n\r\nWhere are the logs (dhcp requests are under /var/log/messages) for the same time in the node that is acting as DHCP server, does that node ever get the DHCP request from the trinity nodes? You should be able to see the request there.\r\n\r\nAlso @hansthen says he can see this in the master node (is that the node running the DHCP server?):\r\n\r\n`[11884.587385] igb: eno2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX\r\n[11885.372575] igb: eno2 NIC Link is Down\r\n[11889.114388] igb: eno2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX\r\n[11890.398844] igb: eno2 NIC Link is Down\r\n[11899.450036] igb: eno2 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX`\r\n\r\nWhy would that interface be toggling rates between 100Mbps and 1000Mbps? That should be seen asap."}}