pyspark crashed at dataframe.collect() with error message "Py4JNetworkError: An error occurred while trying to connect to the Java server"
<p>I have a function my_function which has a for loop iteration. There is a part in the for loop which requires to call a dataframe collect(). It works for the first few loops, but it always crashed at the fifth iteration. Do you guys have any idea why this happens?</p>

<pre><code>  File "my_code.py", line 189, in my_function
    my_df_collect = my_df.collect()
  File "/lib/spark/python/pyspark/sql/dataframe.py", line 280, in collect
    port = self._jdf.collectToPython()
  File "/lib/spark/python/pyspark/traceback_utils.py", line 78, in __exit__
    self._context._jsc.setCallSite(None)
  File "/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 811, in __call__
    answer = self.gateway_client.send_command(command)
  File "/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 624, in send_command
    connection = self._get_connection()
  File "/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 579, in _get_connection
    connection = self._create_connection()
  File "/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 585, in _create_connection
    connection.start()
  File "/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 697, in start
    raise Py4JNetworkError(msg, e)
Py4JNetworkError: An error occurred while trying to connect to the Java server
</code></pre>

<p>Another error message</p>

<pre><code>Exception happened during processing of request from ('127.0.0.1', 55584)
Traceback (most recent call last):
  File "/anaconda/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/anaconda/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/anaconda/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/anaconda/lib/python2.7/SocketServer.py", line 655, in __init__
    self.handle()
  File "/lib/spark/python/pyspark/accumulators.py", line 235, in handle
    num_updates = read_int(self.rfile)
  File "/lib/spark/python/pyspark/serializers.py", line 545, in read_int
    raise EOFError
EOFError
</code></pre>