Fast conditional join in Spark
<p>I am trying to do fast enrichment in Spark with conditional queries.</p>

<p>I have two data sets of key/val: "Event Data" and "Session Map". The "session map" is used to find out who was using a given IP between two timestamps. The "Event data" is a large collection of events, with an IP and a timestamp, which need to be correlated against the "session map" to enrich with username.</p>

<p>Is there an efficient way to enrich the Event Data against Session Map in Spark, or something else?</p>

<h3>Session map:</h3>

<pre><code>(IP, start_time, end_time) -&gt; Name
(192.168.0.l, 2016-01-01 10:00:00, 2016-01-01 22:00:00) -&gt; John
(192.168.0.l, 2016-01-01 22:00:01, 2016-01-02 04:35:00) -&gt; Dana
(10.0.0.12,   2016-01-02 06:00:13, 2016-01-02 09:23:24) -&gt; John
...
</code></pre>

<h3>Event data:</h3>

<pre><code>IP -&gt; timestamp
192.168.0.l, 2016-01-01 10:00:00
192.168.0.l, 2016-01-01 10:00:01
192.168.0.l, 2016-01-01 10:00:02
192.168.0.l, 2016-01-01 10:05:23
...
192.168.0.l, 2016-01-01 22:00:01 
192.168.0.l, 2016-01-01 22:12:35 
192.168.0.l, 2016-01-01 04:12:00
...
</code></pre>