Dividing two columns of a different DataFrames
<p>I am using Spark to do exploratory data analysis on a user log file. One of the analysis that I am doing is average requests on daily basis per host. So in order to figure out the average, I need to divide the total request column of the DataFrame by number unique Request column of the DataFrame.</p>

<pre><code>total_req_per_day_df = logs_df.select('host',dayofmonth('time').alias('day')).groupby('day').count()

avg_daily_req_per_host_df = total_req_per_day_df.select("day",(total_req_per_day_df["count"] / daily_hosts_df["count"]).alias("count"))
</code></pre>

<p>This is what I have written using the PySpark to determine the average. And here is the log of error that I get</p>

<pre><code>AnalysisException: u'resolved attribute(s) count#1993L missing from day#3628,count#3629L in operator !Project [day#3628,(cast(count#3629L as double) / cast(count#1993L as double)) AS count#3630];
</code></pre>

<p>Note: daily_hosts_df and logs_df is cached in the memory. How do you divide the count column of both data frames?</p>