Process data in spark and save the result
<p>I have three tables with data 2Gb  each one.</p>

<ol>
<li>DC - First table contains data of people (name, genere, passport, etc..).</li>
<li>EC - Second table contains addresses of this people.</li>
<li>ECK - The third table contains addresses of bills of this people.</li>
</ol>

<p>I need iterate the second table and search by passport in the third table for generate a score for the best address (multiple algorithms).</p>

<p>How can work with this data and save the result in a table ?</p>

<p><strong>Note: I'm working with parquet files</strong></p>

<hr>

<p><strong><em>Update (@cheseaux): I have done so far</em></strong></p>

<p>I have 3 files: DC.txt, EC.txt and ECK.txt</p>

<p>Import data:</p>

<pre><code>val DC = sqlContext.read
          .format("com.databricks.spark.csv")
          .option("header", "false")
          .option("inferSchema", "false")
          .option("mode", "DROPMALFORMED")
          .option("delimiter", "|")
          .load("/DC.txt")
</code></pre>

<p>and save to hive context:</p>

<pre><code>DC.select("*").write
    .format("com.databricks.spark.csv")
    .option("header", "false")
    .option("inferSchema", "false")
    .option("mode", "DROPMALFORMED")
    .option("delimiter", "|")
    .option("quote", " ")
    .save("hdfs://localhost:8020/project/DC")
</code></pre>

<p>create an external table:</p>

<pre><code>CREATE EXTERNAL TABLE EC(
        some     String, 
        var   String, 
        here     String
    )
ROW FORMAT
DELIMITED FIELDS TERMINATED BY '\|'
STORED AS TEXTFILE 
LOCATION 'hdfs://localhost:8020/project/DC'
</code></pre>

<p>and create a parquet table:</p>

<pre><code>CREATE TABLE 
    EC_PARQUET(
        some     String, 
        var   String, 
        here     String
  )
ROW FORMAT
    SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS
    INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
</code></pre>

<p>Insert all data to parquet table:</p>

<pre><code>INSERT INTO EC_PARQUET SELECT * FROM EC
</code></pre>

<p>Now i'm trying work with this data:</p>

<pre><code>val hc = new org.apache.spark.sql.hive.HiveContext(sc)
val ec = hc.sql("SELECT * FROM EC_PARQUET")
</code></pre>

<p>And trying:</p>

<pre><code>import org.apache.spark.sql.Row;

val test = ec.take(100000).map(row =&gt; {
    val row1 = row.getAs[String](1)
    val make = if (row1.toLowerCase == "cualquiercosa") "S" else row1
    Row(row(0),make,row(3))
  }).registerTempTable("teste2")
</code></pre>

<p>How can create a table with this modification columns ?</p>