Reading S3 data from Google's dataproc
<p>I'm running a pyspark application through Google's dataproc on a cluster I created.  In one stage, the application needs to access a directory in an Amazon S3 directory.  At that stage, I get the error:</p>

<p>AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).</p>

<p>I logged onto the headnode of the cluster and set the /etc/boto.cfg with my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY information, but that didn't solve the access issue.  </p>

<p>(1) Any other suggestions for how to access AWS S3 from a dataproc cluster?  </p>

<p>(2) Also, what is the name of the user that dataproc uses to access the cluster?  If I knew that, I could set the ~/.aws directory on the cluster for that user.</p>

<p>Thanks.</p>