How to find the size or shape of a DataFrame in PySpark?
<p>I am trying to find out the size/shape of a DataFrame in PySpark. I do not see a single function that can do this.</p>
<p>In Python, I can do this:</p>
<pre class="lang-py prettyprint-override"><code>data.shape()
</code></pre>
<p>Is there a similar function in PySpark? This is my current solution, but I am looking for an element one</p>
<pre class="lang-py prettyprint-override"><code>row_number = data.count()
column_number = len(data.dtypes)
</code></pre>
<p>The computation of the number of columns is not ideal...</p>