How can i profile spark application to check time spent by the application in each stage
<p>I am trying to run Performance testing on one of my spark jobs which loads data into memory and then perform spark-sql operations on the data and finally returns the result to user.</p>

<p>I would like to know how much time does the job takes in each stage: i.e. time taken by the spark job to load data into memory and time taken by the job to complete its execution. Is there any good profiling tool available for this.</p>

<p>If not, Can I use already available spark UI/logs to fetch such information.</p>

<p>Or, if there is any other way to get this data.</p>

<p>Please note that I am a newbie in Spark so please excuse for any illogical/irrelevant question.</p>

<p>Thanks in advance!</p>