Running EMR Spark With Multiple S3 Accounts
<p>I have an EMR Spark Job that needs to read data from S3 on one account and write to another.<br>
I split my job into two steps.  </p>

<ol>
<li><p>read data from the S3 (no credentials required because my EMR cluster is in the same account).</p></li>
<li><p>read data in the local HDFS created by step 1 and write it to an S3 bucket in another account.</p></li>
</ol>

<p>I've attempted setting the <code>hadoopConfiguration</code>:</p>

<pre><code>sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "&lt;your access key&gt;")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey","&lt;your secretkey&gt;")
</code></pre>

<p>And exporting the keys on the cluster:</p>

<pre><code>$ export AWS_SECRET_ACCESS_KEY=
$ export AWS_ACCESS_KEY_ID=
</code></pre>

<p>I've tried both <strong>cluster</strong> and <strong>client</strong> mode as well as <strong>spark-shell</strong> with no luck.</p>

<p>Each of them returns an error:</p>

<pre><code>ERROR ApplicationMaster: User class threw exception: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: 
Access Denied
</code></pre>