FP-growth - Items in a transaction must be unique
<p>I have run the code in my computer and with Frequent pattern mining. I use the FP-growth, but the pyspark throws an error and I have no idea how to solve it, so can somebody using the pyspark help me?</p>

<p>First I get the data</p>

<pre><code>data = sc.textFile(somewhere)
</code></pre>

<p>This step has no error
and then</p>

<pre><code>transactions = data.map(lambda line: line.strip().split(' '))
</code></pre>

<p>Next is </p>

<pre><code>model = FPGrowth.train(transactions, minSupport=0.2, numPartitions=10)
</code></pre>

<p>which throws an error</p>

<pre><code>An error occurred while calling o19.trainFPGrowthModel.:org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 3, localhost): org.apache.spark.SparkException: Items in a transaction must be unique but got WrappedArray(,  ,  A,  ,  Seq,  0xBB20C554Ack,  0xE6A8BA01Win,  0x7D78TcpLen,  20).
</code></pre>

<p>My data looks like this</p>

<pre><code> transactions.take(1)

[[u'03/07',
  u' 10',
  u' 22',
  u' 04.439824',
  u' 139',
  u' 1',
  u' 1',
  u' spp_sdf',
  u' SDFCombinationAlert',
  u' Classification',
  u' SenstiveData',
  u' Priority',
  u' 2',
  u' PROTO',
  u' 254',
  u' 197.218.177.69',
  u' 172.16.113.84']]
</code></pre>