Spark Parse Text File to DataFrame
<p>Currently, I can parse a text file to a Spark DataFrame by way of the RDD API with the following code:</p>

<pre><code>def row_parse_function(raw_string_input):
    # Do parse logic...
    return pyspark.sql.Row(...)

raw_rdd = spark_context.textFile(full_source_path)
# Convert RDD of strings to RDD of pyspark.sql.Row
row_rdd = raw_rdd.map(row_parse_function).filter(bool)
# Convert RDD of pyspark.sql.Row to Spark DataFrame.
data_frame = spark_sql_context.createDataFrame(row_rdd, schema)
</code></pre>

<p>Is this current approach ideal?</p>

<p>Or is there a better way to do this without using the older RDD API.</p>

<p>FYI, Spark 2.0.</p>