(Spark/Scala) What would be the most effective way to compare specific data in one RDD to a line of another?
<p>Basically, I have two sets of data in two text files. One set of data is in the format:</p>

<p>a,DataString1,DataString2 (One line) (The first character is in every entry but not relevant)</p>

<p>.... (and so on)</p>

<p>The second set of data is in format:</p>

<p>Data, Data Data Data, Data Data, Data, Data Data Data (One line)(separated by either commas or spaces, but I'm able to use a regular expression to handle this, so that's not the problem)</p>

<p>.... (And so on)</p>

<p>So what I need to do is check if DataString1 AND DataString2 are both present on any single line of the second set of data.</p>

<p>Currently I'm doing this like so:</p>

<pre><code>// spark context is defined above, imported java.util.regex.Pattern above as well
case class test(data_one: String, data_two: String)
// case class is used to just more simply organize data_one to work with
val data_one = sc.textFile("path")
val data_two = sc.textFile("path")

val rdd_one = data_one.map(_.split(",")).map( c =&gt; test(c(1),c(2))
val rdd_two = data_two.map(_.split("[,\\s*]"))
val data_two_array = rdd_two.collect()
// this causes data_two_array to be an array of array of strings.
one.foreach { line =&gt;
    for (array &lt;- data_two_array) {

        for (string &lt;- array) {
            // comparison logic here that checks finds if both dataString1 and dataString2
            // happen to be on same line is in these two for loops
        }
    }
}
</code></pre>

<p>How could I make this process more efficient? At the moment it does work correctly, but as data sizes grow this becomes very ineffective.</p>