{"action":"created","issue":{"url":"https://api.github.com/repos/bloomberg/chef-bach/issues/56","labels_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/56/comments","events_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/56/events","html_url":"https://github.com/bloomberg/chef-bach/issues/56","id":52388639,"number":56,"title":"Namenode disk replacement not automatically handled","user":{"login":"cbaenziger","id":4192761,"avatar_url":"https://avatars.githubusercontent.com/u/4192761?v=3","gravatar_id":"","url":"https://api.github.com/users/cbaenziger","html_url":"https://github.com/cbaenziger","followers_url":"https://api.github.com/users/cbaenziger/followers","following_url":"https://api.github.com/users/cbaenziger/following{/other_user}","gists_url":"https://api.github.com/users/cbaenziger/gists{/gist_id}","starred_url":"https://api.github.com/users/cbaenziger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cbaenziger/subscriptions","organizations_url":"https://api.github.com/users/cbaenziger/orgs","repos_url":"https://api.github.com/users/cbaenziger/repos","events_url":"https://api.github.com/users/cbaenziger/events{/privacy}","received_events_url":"https://api.github.com/users/cbaenziger/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bloomberg/chef-bach/labels/HDFS","name":"HDFS","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-12-18T16:54:16Z","updated_at":"2015-01-06T09:38:55Z","closed_at":null,"body":"When a namenode has a disk replaced, the namenode daemon appears to need a bounce to start repopulating the disk.\r\n\r\nOnce a disk is recovered (e.g. its mount point goes from returning EIO to working again) it appears the namenode will not recognize the disk as recovered and start repopulating it automatically. Our namenode_standby recipe has a [check](https://github.com/bloomberg/chef-bach/blob/master/cookbooks/bcpc-hadoop/recipes/namenode_standby.rb#L43) to ensure all namenode disks have a `current` directory. However, we have two issues:\r\n* Bootstrapping the namenode again requires it to be shutdown (so we need a check to ensure the namenode is stopped before running bootstrap)\r\n* Bootstrapping a namenode requires the edits to be caught up reasonably; rolling the edits can do this (so we need a `dfsadmin -rollEdits`)"},"comment":{"url":"https://api.github.com/repos/bloomberg/chef-bach/issues/comments/68844555","html_url":"https://github.com/bloomberg/chef-bach/issues/56#issuecomment-68844555","issue_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/56","id":68844555,"user":{"login":"cbaenziger","id":4192761,"avatar_url":"https://avatars.githubusercontent.com/u/4192761?v=3","gravatar_id":"","url":"https://api.github.com/users/cbaenziger","html_url":"https://github.com/cbaenziger","followers_url":"https://api.github.com/users/cbaenziger/followers","following_url":"https://api.github.com/users/cbaenziger/following{/other_user}","gists_url":"https://api.github.com/users/cbaenziger/gists{/gist_id}","starred_url":"https://api.github.com/users/cbaenziger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cbaenziger/subscriptions","organizations_url":"https://api.github.com/users/cbaenziger/orgs","repos_url":"https://api.github.com/users/cbaenziger/repos","events_url":"https://api.github.com/users/cbaenziger/events{/privacy}","received_events_url":"https://api.github.com/users/cbaenziger/received_events","type":"User","site_admin":false},"created_at":"2015-01-06T09:38:55Z","updated_at":"2015-01-06T09:38:55Z","body":"From [Hash J, Dec. 9th on CDH-Users](http://grokbase.com/t/cloudera/cdh-user/12c9nfrqqx/trouble-recovering-from-name-node-meta-data-errors-hdfs-namenode-recover-not-working) we can have the namenode attempt namedir recovery at checkpoint via:\r\n\"\"\"\"\r\nP.s. For future, you may enable dfs.namenode.name.dir.restore to true\r\nto not need a restart if a single mount failed and needs to be plugged\r\nback in.\r\n\"\"\"\""}}