Loading CSV into PySpark from S3
<p>Using PySpark I can load a local CSV just fine using this code: </p>

<pre><code>cd ./spark-1.6.0-bin-hadoop2.4/

./bin/pyspark --packages com.databricks:spark-csv_2.11:1.2.0 --driver-memory 4G

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('/my/local/folder/input_data.csv').write.save("/my/local/folder/input_data", format="parquet")
</code></pre>

<p>yet I can't get it to work with a (non-public) CSV stored on S3 because it times out:</p>

<pre><code>sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('s3n://&lt;AWS_ACCESS_KEY_ID&gt;:&lt;AWS_SECRET_ACCESS_KEY&gt;@my.bucket/folder/input_data.csv').write.save("/my/local/folder/input_data", format="parquet")


py4j.protocol.Py4JJavaError: An error occurred while calling o25.load.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.net.SocketException: Operation timed out
</code></pre>

<p>Is this possible and if so, any thoughts what I'm doing wrong? Thanks in advance. </p>