{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/hdfs/issues/245","labels_url":"https://api.github.com/repos/mesosphere/hdfs/issues/245/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/hdfs/issues/245/comments","events_url":"https://api.github.com/repos/mesosphere/hdfs/issues/245/events","html_url":"https://github.com/mesosphere/hdfs/issues/245","id":122608454,"number":245,"title":"Datanode hdfs-site.xml not updated correctly","user":{"login":"yoeduardoj","id":2688777,"avatar_url":"https://avatars.githubusercontent.com/u/2688777?v=3","gravatar_id":"","url":"https://api.github.com/users/yoeduardoj","html_url":"https://github.com/yoeduardoj","followers_url":"https://api.github.com/users/yoeduardoj/followers","following_url":"https://api.github.com/users/yoeduardoj/following{/other_user}","gists_url":"https://api.github.com/users/yoeduardoj/gists{/gist_id}","starred_url":"https://api.github.com/users/yoeduardoj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yoeduardoj/subscriptions","organizations_url":"https://api.github.com/users/yoeduardoj/orgs","repos_url":"https://api.github.com/users/yoeduardoj/repos","events_url":"https://api.github.com/users/yoeduardoj/events{/privacy}","received_events_url":"https://api.github.com/users/yoeduardoj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-16T21:38:59Z","updated_at":"2015-12-17T19:03:41Z","closed_at":null,"body":"While running the hdfs framework on our mesos cluster, we've noticed it stops working frequently. We run on AWS, and have a 10 node mesos cluster, which runs 2 name nodes, 3 journal nodes (2 co-located), and 7 datanodes.\r\n\r\nDatanodes drop off continuously from the file system. We'd have too many logs to post here to show evidence of this issue, but here's what we have noticed:\r\n\r\n- Datanodes drop off. Upon checking, the datanodes that are offline ARE running, but their hdfs-site.xml has an IP of a slave that is now gone from mesos altogether (it was terminated by AWS)\r\n- Checking the framework's config server, the hdfs-site.xml returned by it is correct and has the appropriate name node configuration.\r\n- Checking the stdout/stderr of the datanodes that have dropped off, there is a claim that the configuration is being refreshed from the config server. But checking /opt/mesosphere/hdfs/etc/hdfs-site.xml or the sandbox for the datanode reveals the file is NOT being updated.\r\n\r\nTried killing the data node process, but that didn't seem to work, with the hdfs-site.xml still out of date. Killing the executor process and the data node process at the same time, then the file started getting updated correctly, and the datanode(s) came up.\r\n\r\nWe also observed the same issue with the journal nodes and name nodes. With the name nodes, there's an added issue. If both name nodes are killed at the same time, they can never come up, since the configs are updated incorrectly. For example, in that situation, a backup name node doesn't exist,  but the process still puts out an hdfs-site.xml with the configuration for the secondary name node, which just contains :port. Then, both primary and secondary name nodes fail to come up because of an invalid pdfs-site.xml."},"comment":{"url":"https://api.github.com/repos/mesosphere/hdfs/issues/comments/165549843","html_url":"https://github.com/mesosphere/hdfs/issues/245#issuecomment-165549843","issue_url":"https://api.github.com/repos/mesosphere/hdfs/issues/245","id":165549843,"user":{"login":"bhallakapil","id":294028,"avatar_url":"https://avatars.githubusercontent.com/u/294028?v=3","gravatar_id":"","url":"https://api.github.com/users/bhallakapil","html_url":"https://github.com/bhallakapil","followers_url":"https://api.github.com/users/bhallakapil/followers","following_url":"https://api.github.com/users/bhallakapil/following{/other_user}","gists_url":"https://api.github.com/users/bhallakapil/gists{/gist_id}","starred_url":"https://api.github.com/users/bhallakapil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bhallakapil/subscriptions","organizations_url":"https://api.github.com/users/bhallakapil/orgs","repos_url":"https://api.github.com/users/bhallakapil/repos","events_url":"https://api.github.com/users/bhallakapil/events{/privacy}","received_events_url":"https://api.github.com/users/bhallakapil/received_events","type":"User","site_admin":false},"created_at":"2015-12-17T19:03:41Z","updated_at":"2015-12-17T19:03:41Z","body":"Adding a bit more here. It appears that config server IP & port available in executorInfo and is not updated once hdfs scheduler restarts on different node. Only way to recover is to restart Executor so that it gets correct host:port for config server. It may be better to store config server host:port information in zookeeper or pass it in reloadConfigsOnAllRunningTasks call"}}