Efficiently filter ordered file with spark
<p>Imagine I have a big log file of the format:</p>

<pre><code>Timestamp, text
Timestamp, text
Timestamp, text
Timestamp, text
Timestamp, text
Timestamp, text
</code></pre>

<p>The log file is already ordered by timestamp.
If I read the file with spark as a dataframe and then filter on timestamp between t1 and t2,
Then he checks for every record if the timestamp is between t1 and t2, which takes a lot of time.</p>

<p>However, is there a way that I can tell spark that the df is already ordered, then it would know it would just have to look for the first and last timestamp, and return all the lines in between, which would be a lot faster?</p>