Spark DataFrame column name case sensitivity in sparkSQL and Spark Submit
<p>When i am querying dataframes on spark-shell(1.6 version) ,the column names are case insensitive .
On Spark-Shell</p>

<pre><code> val a = sqlContext.read.parquet("&lt;my-location&gt;")
   a.filter($"name" &lt;=&gt; "andrew").count()
   a.filter($"NamE" &lt;=&gt; "andrew").count()
</code></pre>

<p>Both the above results gives me the right count.
But when i build this in a jar and run via "spark-submit",below code fails saying NamE does not exist,since underlying parquet data was saved with column as "name"</p>

<p>Fails:</p>

<pre><code>a.filter($"NamE" &lt;=&gt; "andrew").count()
</code></pre>

<p>Pass:</p>

<pre><code>a.filter($"name" &lt;=&gt; "andrew").count()
</code></pre>

<p>Am i missing something here?is there a way i can make it case-insensitive.
I know i can use a select before filtering and make all columns as lowercase alias ,but wanted to know why is it behaving differently.</p>