How to Store a Python bytestring in a Spark Dataframe
<p>I'm looking at a mystery. I have a bunch of long documents available as Python bytestrings (<code>b"I'm a byte string"</code>) in a RDD. Now I convert this RDD to a <code>DataFrame</code> to join it to another <code>DataFrame</code>. I do that like this:</p>

<pre><code>Data_RDD = Paths_RDD.map(open_paths).flatMap(split_files)

Data_schema = StructType([
    StructField("URI", StringType(), True),
    StructField("Content", StringType(), True),
])

Data_DF = sqlContext.createDataFrame(Data_RDD, schema=Data_schema)

print(Data_DF.show(5))

+--------------------+-----------+
|                 URI|    Content|
+--------------------+-----------+
|http://01storytel...|[B@10628e42|
|http://05yxgs.com...|[B@36699775|
|http://1.lhcmaima...|[B@4e569e3b|
|http://100100.ove...|[B@18ae5bab|
|http://1015theriv...|[B@5f044435|
+--------------------+-----------+
only showing top 5 rows
</code></pre>

<p>these short <code>"[B@10628e42"</code> strings seem fairly useless to me and are probably some kind of pointer. The bytestrings are still 'intact' in the RDD because I can still access them. So in the conversion from RDD to <code>DataFrame</code> the problem occurs. Now I tried to store the bytestrings in fields with other types, namely <code>ByteType()</code> and <code>BinaryType()</code>. Both not working because the bytestrings are not accepted with these error messages:</p>

<pre><code>TypeError: ByteType can not accept object b'some string' in type &lt;class 'bytes'&gt;
TypeError: BinaryType can not accept object b'some string' in type &lt;class 'bytes'&gt;
</code></pre>

<p>But it gets even weirder. When I setup a small scale experiment:</p>

<pre><code>ByteStrings = [b'one',b'two',b'three']
rdd_ByteStrings = sc.parallelize(ByteStrings)
print(rdd_ByteStrings.take(3))

DF2_schema = StructType([
    StructField("ByteString", StringType(), True),
])
DF_ByteStrings = sqlContext.createDataFrame(rdd_ByteStrings,schema=DF2_schema)

print(DF_ByteStrings.show())
</code></pre>

<p>The small bytestrings are not allowed as in a StringType column as well. When I try to run this I get this error message:</p>

<pre><code>StructType can not accept object b'one' in type &lt;class 'bytes'&gt;
</code></pre>

<p>when I try to let spark infer a type it also fails with this message:</p>

<pre><code>TypeError: Can not infer schema for type: &lt;class 'bytes'&gt;
</code></pre>

<p>So any ideas how I could store bytestrings in a <code>DataFrame</code> without to <code>.decode()</code> them. That is something I only can do after I joined the two <code>DataFrames</code> together, because the other one holds the decoding information.</p>

<p>I use Python 3.5 and Spark 2.0.1</p>

<p>Thanks in advance!</p>