Apache Spark-Get All Field Names From Nested Arbitrary JSON Files
<p>I have run into a somewhat perplexing issue that has plagued me for several months now.  I am trying to create an Avro Schema (schema-enforced format for serializing arbitrary data, basically, as I understand it) to convert some complex JSON files (arbitrary and nested) eventually to Parquet in a pipeline.</p>

<p>I am wondering if there is a way to get the superset of field names I need for this use case staying in Apache Spark instead of Hadoop MR in a reasonable fashion?</p>

<p>I think Apache Arrow under development might be able to help avoid this by treating JSON as a first class citizen eventually, but it is still aways off yet.</p>

<p>Any guidance would be sincerely appreciated!</p>