Spark SQL - READ and WRITE in sequence or pipeline?
<p>I am working on a cost function for Spark SQL.
While modelling the TABLE SCAN behaviour I cannot understand if READ and WRITE are carried out in pipeline or in sequence.</p>

<p>Let us consider the following SQL query:</p>

<p>SELECT * FROM table1 WHERE columnA = ‘xyz’;</p>

<p>Each task:</p>

<ol>
<li>Reads a data block (either locally or from a remote node)</li>
<li>Filter out the tuples that do not satisfy the predicate</li>
<li>Write to the disk the remaining tuples</li>
</ol>

<p>Are (1), (2) and (3) carried out in sequence or in pipeline? In other words, the data block is completely read (all the disk pages composing it) first and then it is filtered and then it is rewritten to the disk or are these activities carried out in pipeline? (i.e. while reading the (n+1)-tuple, n-tuple can be processed and written).</p>

<p>Thanks in advance.</p>