Apply a function to a single column of a csv in Spark
<p>Using Spark I'm reading a csv and want to apply a function to a column on the csv. I have some code that works but it's very hacky. What is the proper way to do this?</p>

<p>My code</p>

<pre><code>SparkContext().addPyFile("myfile.py")
spark = SparkSession\
    .builder\
    .appName("myApp")\
    .getOrCreate()
from myfile import myFunction

df = spark.read.csv(sys.argv[1], header=True,
    mode="DROPMALFORMED",)
a = df.rdd.map(lambda line: Row(id=line[0], user_id=line[1], message_id=line[2], message=myFunction(line[3]))).toDF()
</code></pre>

<p>I would like to be able to just call the function on the column name instead of mapping each row to <code>line</code> and then calling the function on <code>line[index]</code>.</p>

<p>I'm using Spark version 2.0.1</p>