Is it possible to do near real-time joins with Spark Streaming ?
<p>So I am using Spark and Spark Streaming for log analytics and Elasticsearch for search and storage. My goal is to associate a score to each session (a set of logs with the same id) and then add those score to each log record. The first part is done easily with Spark Streaming's <code>updateStateByKey</code>, the second part however seems to be impossible at near real-time. So I would like to know if there is a way to perform those joins with the same technologies I am using, or if there is another way around with other solutions. </p>