Select not null values in dataframes in Spark
<p>I'm reading a CSV file in Spark 2.0 and counting not null values in a column using the following:</p>

<pre><code>val df = spark.read.option("header", "true").csv(dir)

df.filter("IncidntNum is not null").count()
</code></pre>

<p>and it works fine when I test it using spark-shell. When I create a jar file containing the code and submit it to spark-submit, I get an exception at second line above :</p>

<pre><code>Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input '' expecting {'(', 'SELECT', ..
== SQL ==
IncidntNum is not null
^^^

        at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:197)
</code></pre>

<p>Any idea why this would happen when I'm using the code that works in spark-shell?</p>