Reading Avro messages from Kafka with Spark 2.0.2 (structured streaming)
<p>I have a spark 2.0 application that reads messages from kafka using spark streaming (with spark-streaming-kafka-0-10_2.11).</p>

<p>Structured streaming looks really cool so I wanted to try and migrate the code but I can't figure out how to use it.</p>

<p>in the regular streaming I used kafkaUtils to createDstrean and in the parameters I passed it was the value deserializer.</p>

<p>in the Structured streaming the doc says that I should deserialize using DataFrame functions but I can't figure exactly what that means.</p>

<p>I looked at examples such as this <a href="https://docs.databricks.com/spark/latest/structured-streaming/kafka.html" rel="nofollow noreferrer">example</a> but my Avro object in Kafka is quit complex and cannot be simply casted like the String in the example..</p>

<p>So far I tried this kind of code (which I saw here in a different question):</p>

<pre><code>import spark.implicits._

  val ds1 = spark.readStream.format("kafka").
    option("kafka.bootstrap.servers","localhost:9092").
    option("subscribe","RED-test-tal4").load()

  ds1.printSchema()
  ds1.select("value").printSchema()
  val ds2 = ds1.select($"value".cast(getDfSchemaFromAvroSchema(Obj.getClassSchema))).show()  
  val query = ds2.writeStream
    .outputMode("append")
    .format("console")
    .start()
</code></pre>

<p>and I get "data type mismatch: cannot cast BinaryType to StructType(StructField(...."</p>

<p>how can I deserialize the value?</p>