New data not picked up from S3 in Spark Structured Streaming
<p>I am trying to read data from an S3 bucket in Spark Structured Streaming. The below code works to get existing data. But, when new data is added to the bucket, Spark does not pick this. </p>

<pre class="lang-scala prettyprint-override"><code>val lines = spark.readStream.schema(schemaImp).format("com.databricks.spark.avro").load("s3n://bucket/*")
val query = lines.writeStream.outputMode("append").format("memory").queryName("memtable").start()
query.processAllAvailable()
spark.sql("select * from memtable").show()
</code></pre>

<p>How can I make this work to fetch new data? Or, is this a feature that's not supported yet? </p>