Using s3a to fetch spark jars in driver and executor
<p>I'm trying to use Spark on AWS using the driver/executor model. There doesn't seem to be a way to set the driver classpath such that it can use the hadoop-aws jar plus the aws-sdk jar to get access to s3. I want to pull my job jars from s3. </p>

<p>Seems like I either need to add all these classes to the spark assembly jar, or I need to modify the spark-class script to add these jars to the classpath manually?</p>

<p>Are there any less intrusive methods, i.e. any way to just specify this in config somewhere?</p>

<p>I'm running 1.6.1 and hadoop 2.4, but I expect I can use 2.6 too (same problem).</p>