Create new column with function in Spark Dataframe based on a string search of another column
<p>I have a spark dataframe with a column that contains string values (i.e. 'xyztext\afadfa'). I wish to create a new column where the values are '0' or '1' depending on whether the original column contains certain text (i.e. 'text')</p>

<p>Example of result:</p>

<pre><code>## +---+---+------+---------+
## | x1| x2|   x3 | xnew    |
## +---+---+------+---------+
## |  1|  a| xtext|    1    |
## |  3|  B| abcht|    0    |
</code></pre>

<p>EDIT: I have tried this previously (and have now added .cast(int)) thanks to SGVD but receive 'column is not callable' error when I insert the column name:
    df1 = df.withColumn('Target', df.column.contains('text').cast('int'))</p>

<p>The best I have achieved so far is creating a column with 0's in it by:</p>

<pre><code>from pyspark.sql.functions import lit
df1 = df.withColumn('Target', lit(0))
</code></pre>

<p>I have also tried an if then else statement to create the vector but am having no luck:</p>

<pre><code> def targ(string):
     if df.column.contains('text'): return '1'
     else: return '0'
</code></pre>