IllegalArgumentException, Wrong FS when writing ML model to s3 from Spark (Scala)
<p>I've created a model:</p>

<pre><code>val model = pipeline.fit(commentLower)
</code></pre>

<p>and I'm attempting to write it to s3:</p>

<pre><code>sc.hadoopConfiguration.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
sc.hadoopConfiguration.set("fs.s3.awsAccessKeyId", "MYACCESSKEY")
sc.hadoopConfiguration.set("fs.s3.awsSecretAccessKey", "MYSECRETKEY")
model.write.overwrite().save("s3n://sparkstore/model")
</code></pre>

<p>but I'm getting this error:</p>

<pre><code>Name: java.lang.IllegalArgumentException
Message: Wrong FS: s3n://sparkstore/model, expected: file:///
StackTrace: org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80)
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529)
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:80)
</code></pre>

<p>I also tried with my access key inline:</p>

<pre><code>model.write.overwrite().save("s3n://MYACCESSKEY:MYSECRETKEY@/sparkstore/model")
</code></pre>

<p>How do I write a model (or any file for that matter) to s3 from Spark?</p>