Spring boot Hadoop, Webhdfs and Apache Knox
<p>I have a Spring boot application which is accessing HDFS through Webhdfs secured via Apache Knox secured by Kerberos. I created my own <code>KnoxWebHdfsFileSystem</code> with custom scheme (<em>swebhdfsknox</em>) as a subclass of <code>WebHdfsFilesystem</code> which only changes the URLs to contain the Knox proxy prefix. So it effectively remaps requests from form:</p>

<p><code>http://host:port/webhdfs/v1/...</code> </p>

<p>to the Knox one:</p>

<p><code>http://host:port/gateway/default/webhdfs/v1/...</code></p>

<p>I do this by overriding two methods:</p>

<ol>
<li><code>public URI getUri()</code></li>
<li><code>URL toUrl(Op op, Path fspath, Param&lt;?, ?&gt;... parameters)</code></li>
</ol>

<p>So far so good. I let spring boot create <code>FsShell</code> for me and use it for various operations such as <em>list files</em>, <em>mkdir</em> etc. All work fine. Except <em>copyFromLocal</em> which as documented requires 2 steps and redirect. And on the last step when the filesystem tries to <code>PUT</code> to the final URL which received in <em>Location</em> header it fails with error:</p>

<pre><code>org.apache.hadoop.security.AccessControlException: Authentication required
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:334) ~[hadoop-hdfs-2.6.0.jar:na]
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) ~[hadoop-hdfs-2.6.0.jar:na]
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathOutputStreamRunner$1.close(WebHdfsFileSystem.java:787) ~[hadoop-hdfs-2.6.0.jar:na]
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:54) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:302) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1889) ~[hadoop-common-2.6.0.jar:na]
    at org.springframework.data.hadoop.fs.FsShell.copyFromLocal(FsShell.java:265) ~[spring-data-hadoop-core-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.data.hadoop.fs.FsShell.copyFromLocal(FsShell.java:254) ~[spring-data-hadoop-core-2.2.0.RELEASE.jar:2.2.0.RELEASE]
</code></pre>

<p>I suspect the problem is the redirect somehow but can't figure out what might be the problem here. If I do the same requests via <em>curl</em> the file is successfully uploaded to HDFS.</p>