Iterate through S3 files in Spark
<p>Problem: <br>
Large number of files. Each file is 10MB and consist of records in json format, gzipped. </p>

<p>My snippet is loading all the data into memory. There is no need to do this. I just need a few hours of data in memory at a time. I need a sliding window.</p>

<p>Is it possible to apply the 'window' idea from spark streaming to the files and how would I do this? </p>

<p>I'm using python</p>

<pre><code>location = "s3://bucketname/xxxx/2016/10/1[1-2]/*/file_prefix*.gz"
rdd = sc.textFile(location)
</code></pre>