Count number of elements in each pyspark RDD partition
<p>I'm looking for the Pyspark equivalent to this question: <a href="https://stackoverflow.com/questions/28687149/how-to-get-the-number-of-elements-in-partition/28690918#28690918">How to get the number of elements in partition?</a>.</p>

<p>Specifically, I want to programmatically count the number of elements in each partition of a pyspark RDD or dataframe (I know this information is available in the Spark Web UI). </p>

<p>This attempt: </p>

<p><code>df.foreachPartition(lambda iter: sum(1 for _ in iter))</code></p>

<p>results in: </p>

<blockquote>
  <p>AttributeError: 'NoneType' object has no attribute '_jvm'</p>
</blockquote>

<p>I do not want to collect the contents of the iterator into memory.</p>