Using Spark Context To Read Parquet File as RDD(wihout using Spark-Sql Context) giving Exception
<p>I am trying to read and write Parquet file as RDD using Spark. I cant use Spark-Sql-Context in my current application(It needs a parquet schema in StructType which when I convert from Avro Schema gives me castException in few cases) </p>

<p>So if i try to implement and save Parquet File by overload AvroParquetFormat and Sending ParquetInputFormat to Hadoop To write in following way: </p>

<pre><code>def saveAsParquetFile[T &lt;: IndexedRecord](records: RDD[T], path: String)(implicit m: ClassTag[T]) = {
            val keyedRecords: RDD[(Void, T)] = records.map(record =&gt; (null, record))
            spark.hadoopConfiguration.setBoolean("parquet.enable.summary-metadata", false)
            val job = Job.getInstance(spark.hadoopConfiguration)
            ParquetOutputFormat.setWriteSupportClass(job, classOf[AvroWriteSupport])
            AvroParquetOutputFormat.setSchema(job, m.runtimeClass.newInstance().asInstanceOf[IndexedRecord].getSchema())
            keyedRecords.saveAsNewAPIHadoopFile(
              path,
              classOf[Void],
              m.runtimeClass.asInstanceOf[Class[T]],
              classOf[ParquetOutputFormat[T]],
              job.getConfiguration
            )
          }
</code></pre>

<p>This is thowing error:</p>

<pre><code>Exception in thread "main" java.lang.InstantiationException: org.apache.avro.generic.GenericRecord
</code></pre>

<p>I am calling The function as follows:</p>

<pre><code>val file1: RDD[GenericRecord] = sc.parquetFile[GenericRecord]("/home/abc.parquet")
sc.saveAsParquetFile(file1, "/home/abc/")
</code></pre>