{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/1379","labels_url":"https://api.github.com/repos/druid-io/druid/issues/1379/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1379/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/1379/events","html_url":"https://github.com/druid-io/druid/pull/1379","id":79015455,"number":1379,"title":"bug fix: hdfs task log and indexing task not work properly with Hadoop HA","user":{"login":"flowbehappy","id":2639319,"avatar_url":"https://avatars.githubusercontent.com/u/2639319?v=3","gravatar_id":"","url":"https://api.github.com/users/flowbehappy","html_url":"https://github.com/flowbehappy","followers_url":"https://api.github.com/users/flowbehappy/followers","following_url":"https://api.github.com/users/flowbehappy/following{/other_user}","gists_url":"https://api.github.com/users/flowbehappy/gists{/gist_id}","starred_url":"https://api.github.com/users/flowbehappy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flowbehappy/subscriptions","organizations_url":"https://api.github.com/users/flowbehappy/orgs","repos_url":"https://api.github.com/users/flowbehappy/repos","events_url":"https://api.github.com/users/flowbehappy/events{/privacy}","received_events_url":"https://api.github.com/users/flowbehappy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-05-21T14:20:31Z","updated_at":"2015-05-22T12:41:49Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/1379","html_url":"https://github.com/druid-io/druid/pull/1379","diff_url":"https://github.com/druid-io/druid/pull/1379.diff","patch_url":"https://github.com/druid-io/druid/pull/1379.patch"},"body":"Fix:\r\n\r\n* While using HDFS with Hadoop HA to store task logs,  exception could be thrown while fetching or pushing task log:\r\n\r\n```\r\n2015-05-22T11:05:44,987 [WARN ] [qtp164455471-95] io.druid.indexing.overlord.http.OverlordResource - Failed to stream log for task index_hadoop_dsp_campaign_only_2015-05-21T12:44:10.075Z\r\njava.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster\r\n\tat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:258) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:153) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:602) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:547) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:139) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) ~[hadoop-common-2.5.1.jar:?]\r\n\tat io.druid.storage.hdfs.tasklog.HdfsTaskLogs.streamTaskLog(HdfsTaskLogs.java:65) ~[?:?]\r\n\tat io.druid.indexing.common.tasklogs.SwitchingTaskLogStreamer.streamTaskLog(SwitchingTaskLogStreamer.java:46) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat io.druid.indexing.overlord.http.OverlordResource.doGetLog(OverlordResource.java:382) [druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_67]\r\n```\r\n\r\nand\r\n\r\n```\r\n2015-05-22T11:19:00,804 [INFO ] [pool-10-thread-2] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[0] for task: index_hadoop_dsp_campaign_only_2015-05-22T03:18:46.860Z\r\n2015-05-22T11:19:00,804 [INFO ] [pool-10-thread-2] io.druid.storage.hdfs.tasklog.HdfsTaskLogs - Writing task log to: hdfs://mycluster/druid/tasklogs/index_hadoop_dsp_campaign_only_2015-05-22T03_18_46.860Z\r\n2015-05-22T11:19:00,882 [INFO ] [pool-10-thread-2] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution\r\njava.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster\r\n\tat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:258) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:153) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:602) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:547) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:139) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) ~[hadoop-common-2.5.1.jar:?]\r\n\tat io.druid.storage.hdfs.tasklog.HdfsTaskLogs.pushTaskLog(HdfsTaskLogs.java:56) ~[?:?]\r\n\tat io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:244) [druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:118) [druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_67]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_67]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_67]\r\n\tat java.lang.Thread.run(Thread.java:745) [?:1.7.0_67]\r\nCaused by: java.net.UnknownHostException: mycluster\r\n\t... 19 more\r\n```\r\n\r\n* While using HDFS with Hadoop HA as deep storage, could throw exceptions while running indexing job:\r\n\r\n```\r\nCaused by: java.net.UnknownHostException: mycluster\r\n\tat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:258) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:153) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:602) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:547) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:139) ~[hadoop-hdfs-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) ~[hadoop-common-2.5.1.jar:?]\r\n\tat org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518) ~[hadoop-mapreduce-client-core-2.5.1.jar:?]\r\n\tat org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPaths(FileInputFormat.java:483) ~[hadoop-mapreduce-client-core-2.5.1.jar:?]\r\n\tat io.druid.indexer.path.StaticPathSpec.addInputPaths(StaticPathSpec.java:56) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat io.druid.indexer.HadoopDruidIndexerConfig.addInputPaths(HadoopDruidIndexerConfig.java:316) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat io.druid.indexer.JobHelper.ensurePaths(JobHelper.java:121) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:53) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\tat io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:334) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]\r\n\t... 11 more\r\n``` \r\n\r\n* Make indexing worker, i.e. Peon process also inherit middleManager's  Java args with `hadoop` prefix.\r\n\r\nArgs like this:\r\n```\r\n-Dhadoop.fs.defaultFS=hdfs://mycluster \\\r\n-Dhadoop.dfs.nameservices=mycluster \\\r\n-Dhadoop.dfs.ha.namenodes.mycluster=nn1,nn2 \\\r\n-Dhadoop.dfs.namenode.rpc-address.mycluster.nn1=hadoop.namenode1:9000 \\\r\n-Dhadoop.dfs.namenode.rpc-address.mycluster.nn2=hadoop.namenode2:9000 \\\r\n-Dhadoop.dfs.client.failover.proxy.provider.mycluster=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/104650068","html_url":"https://github.com/druid-io/druid/pull/1379#issuecomment-104650068","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1379","id":104650068,"user":{"login":"flowbehappy","id":2639319,"avatar_url":"https://avatars.githubusercontent.com/u/2639319?v=3","gravatar_id":"","url":"https://api.github.com/users/flowbehappy","html_url":"https://github.com/flowbehappy","followers_url":"https://api.github.com/users/flowbehappy/followers","following_url":"https://api.github.com/users/flowbehappy/following{/other_user}","gists_url":"https://api.github.com/users/flowbehappy/gists{/gist_id}","starred_url":"https://api.github.com/users/flowbehappy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flowbehappy/subscriptions","organizations_url":"https://api.github.com/users/flowbehappy/orgs","repos_url":"https://api.github.com/users/flowbehappy/repos","events_url":"https://api.github.com/users/flowbehappy/events{/privacy}","received_events_url":"https://api.github.com/users/flowbehappy/received_events","type":"User","site_admin":false},"created_at":"2015-05-22T12:41:49Z","updated_at":"2015-05-22T12:41:49Z","body":"@sixtus that's weird it doesn't work for me :( \r\nBesides even if it works, I still think this bug is worthy to fix for someone who chose to set hadoop config in that way."}}