Spark DataFrame - .distinct() not working?
<p>I'm using the following code: </p>

<pre><code>df = df.select(
            df.col("col").as("col1"),
            df.col("col_").as("col2");
df = df.select("*").distinct();
df= df.sample(true, 0.8).limit(300);
df= df.withColumn("random", lit(0));

df.show();
</code></pre>

<p>I want to select distinct rows, then take a sample and limit it to 300 records, however <code>df.show();</code> shows that there are duplicate rows all over the place. What am I missing?</p>

<p>Thank you!</p>