Partition RDD in Apache Spark such that one partition consists on one file
<p>I am creating a single RDD of 2.csv files like this </p>

<pre><code>val combineRDD = sc.textFile("D://release//CSVFilesParellel//*.csv")
</code></pre>

<p>Then I want to define custom partition on this RDD such that one partition must contain one file.
so that each partition i.e.one csv file is processed across one node for faster data processing </p>

<p>Is it possible to write a custom partitioner based on the file size or the number of lines in one file or the end of file character of one file ? </p>

<p>How do I achieve this ?</p>

<p>The structure of one file looks something like this:</p>

<p>00-00</p>

<p>Time(in secs)  Measure1  Measure2  Measure3..... Measuren</p>

<p>0   </p>

<p>0.25</p>

<p>0.50</p>

<p>0.75</p>

<p>1</p>

<p>...</p>

<p>3600</p>

<hr>

<p>1.The first row of data contains the hours: mins 
    Each file contains data for 1 hour or 3600secs</p>

<p>2.The first column is a second divided into 4 parts of 250 ms each and data recorded for 250 ms </p>

<ol start="3">
<li><p>For every file I want to add the number of hours: mins to the seconds so that my time looks something like this hours-mins-secs. But the catch is I dont want this process to happen sequentially </p></li>
<li><p>I am using  the for-each function for getting each file name -> then creating an RDD of the data in the file and adding the time as specified above.</p></li>
<li><p>But what I want is that <strong>every file</strong> should go to one node for processing and calculating time as opposed <strong>to data in one file</strong> getting distributed across nodes for calculating the time.</p></li>
</ol>

<p>Thank you.</p>

<p>Regards,</p>

<p>Vinay Joglekar</p>