Spark : SQL Context : Creating a data frame from Scala object
<p>I am using Spark 1.5.2 to create a data frame from scala object using of of the following syntax. My purpose is to create a data for for unit testing.</p>



<pre class="lang-scala prettyprint-override"><code>class Address (first:String = null, second: String = null, zip: String = null){}
class Person (id: String = null, name: String = null, address: Seq[Address] = null){}
</code></pre>

<hr>

<pre class="lang-scala prettyprint-override"><code>def test () = {

  val sqlContext = new SQLContext(sc)

  import sqlContext.implicits._

  val persons = Seq(
    new Person(id = "1", name = "Salim", 
      address = Seq(new Address(first = "1st street"))),
    new Person(name = "Sana",
      address = Seq(new Address(zip = "60088")))
  )

  // The code can't infer schema automatically
  val claimDF = sqlContext.createDataFrame(sc.parallelize(persons, 2),classOf[Person])

  claimDF.printSchema() // This prints "root" not the schema of Person.
}
</code></pre>

<p><strong>Instead if I convert the Person and Address to case class then Spark can inherit schema automatically</strong> using the above syntax or using <code>sc.parallelize(persons, 2).toDF</code> or using <code>sqlContext.createDataFrame(sc.parallelize(persons, 2),StructType)</code></p>

<p>I can't use case class because it can't hold more than 20 fields and I have a lot of fields in the class. And using StructType causes a lot of inconvenience. Case class is most convenient but can't hold too many properties.</p>

<p>Please help, thanks in advance.</p>