Scala ElasticSearch indexing in relation to dynamically changing schema
<p>I found the following code below  on CERN's website.
FYI: I am using spark 1.3
The example code is fantastic  when you know the schema of the dataset you want to index to elasticsearch.</p>

<p>However, could somebody point me in the right direction so that I can create a method as follows:</p>

<p>Pass in as an argument  the schema structure from external source (col name / datatype) (the hard bit) along with file name to be indexed (easy bit)?
Perform the schema mappings inside the function dynamically.</p>

<p>By having a method like this would allow me to generate a mapped and indexed dataset in ES.</p>

<p>Example Code:</p>

<pre><code>    //import elasticsearch packages
    import org.elasticsearch.spark._

    //define the schema
    case class MemT(dt: String (link is external), server: String (link is external), memoryused: Integer (link is external)) 

    //load the csv file into rdd
    val Memcsv = sc.textFile("/tmp/flume_memusage.csv") 

    //split the fields, trim and map it to the schema
    val MemTrdd = Memcsv.map(line=&gt;line.split(",")).map(line=&gt;MemT(line(0).trim.toString,line(1).trim.toString,line(2).trim.toInt))

    //write the rdd to the elasticsearch
    MemTrdd.saveToEs("fmem/logs")
</code></pre>

<p>Thank you!</p>

<p>source:
<a href="https://db-blog.web.cern.ch/blog/prasanth-kothuri/2016-05-integrating-hadoop-and-elasticsearch-%E2%80%93-part-2-%E2%80%93-writing-and-querying" rel="nofollow">https://db-blog.web.cern.ch/blog/prasanth-kothuri/2016-05-integrating-hadoop-and-elasticsearch-%E2%80%93-part-2-%E2%80%93-writing-and-querying</a></p>