Spark Streaming Multiple Streaming Jobs
<p>I have an application where multiple users are able to specify Spark workflows which are then send to the driver and executed on the cluster. </p>

<p>The workflows should now be extended to also support streamed data-sources. A possible workflow could involve:</p>

<ol>
<li>Stream tweets with a specific hashtag</li>
<li>Transform each tweet</li>
<li>Do analysis on a windowed frame and visualization</li>
</ol>

<p>This is working if only one single stream is started at once but gives the "<em>Only one StreamingContext may be started in this JVM.</em>" error.</p>

<p>I tried different known approaches but none of them was working for me ("spark.driver.allowMultipleContexts = true", increasing "spark.streaming.concurrentJobs", trying to run each streaming context in a different pool, etc.)</p>

<p><strong>Can anybody tell me what the current best practice regarding parallel streams  with Spark streaming is?</strong>
Thx in advance!</p>