How to time a transformation in Spark, given lazy execution style?
<p>I have some code that is executing quite a few steps, and I know how long the whole process takes. However, I would like to be able to calculate how long each individual transformation takes. Here are some simple examples of the steps:</p>

<pre><code>rdd1 = sc.textFile("my/filepath/*").map(lambda x: x.split(","))
rdd2 = sc.textFile("other/filepath/*").map(lambda x: x.split(","))
to_kv1 = rdd1.map(lambda x: (x[0], x[1])) # x[0] is key, x[1] is numeric
to_kv2 = rdd2.map(lambda x: (x[0], x[1])) # x[0] is key, x[1] is numeric
reduced1 = to_kv1.reduceByKey(lambda a, b: a+b)
reduced2 = to_kv1.reduceByKey(lambda a, b: a+b)
outer_joined = reduced1.fullOuterJoin(reduced2) # let's just assume there is key overlap
outer_joined.saveAsTextFile("my_output")
</code></pre>

<p>Now: how do I benchmark a specific part of this code? I know that running it end to end takes a certain amount of time (the <code>saveAsTextFile</code> will force it to execute), but how do I benchmark only the <code>reduceByKey</code> or <code>fullOuterJoin</code> part of the code? I know I could run <code>count()</code> after each operation to force execution, but that would not properly benchmark the operation since it is adding the time it takes to perform the <code>count</code> as well as the time to perform the transformation.</p>

<p>What is the best way to benchmark Spark transformations, given their lazy execution style? </p>

<p>Please note that I am NOT asking how to measure time. I know about the <code>time</code> module, <code>start = time.time()</code>, etc. I am asking how to benchmark given the lazy execution style of Spark transformations, which do not execute until you call an action that requires information to be returned to the driver.</p>