Dealing with a large gzipped file in Spark
<p>I have a large (about 85 GB compressed) gzipped file from s3 that I am trying to process with Spark on AWS EMR (right now with an m4.xlarge master instance and two m4.10xlarge core instances each with a 100 GB EBS volume).  I am aware that gzip is a non-splittable file format, and <a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-rdd-partitions.html#repartitioning" rel="noreferrer">I've</a> <a href="https://forums.databricks.com/questions/113/why-does-my-job-only-launch-a-small-number-1-numbe.html" rel="noreferrer" title="seen">seen</a> <a href="https://stackoverflow.com/questions/16302385/gzip-support-in-spark#comment34852618_16309699">it</a> <a href="http://snappishproductions.com/2015/09/28/Troubleshooting-Apache-Spark-GZip.html" rel="noreferrer">suggested</a> that one should repartition the compressed file because Spark initially gives an RDD with one partition.  However, after doing </p>

<pre><code>scala&gt; val raw = spark.read.format("com.databricks.spark.csv").
     | options(Map("delimiter" -&gt; "\\t", "codec" -&gt; "org.apache.hadoop.io.compress.GzipCodec")).
     | load("s3://path/to/file.gz").
     | repartition(sc.defaultParallelism * 3)
raw: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [_c0: string, _c1: string ... 48 more fields
scala&gt; raw.count()
</code></pre>

<p>and taking a look at the Spark application UI, I still see only one active executor (the other 14 are dead) with one task, and the job never finishes (or at least I've not waited long enough for it to). </p>

<ul>
<li>What is going on here?  Can someone help me understand how Spark is working in this example?  </li>
<li>Should I be using a different cluster configuration?  </li>
<li>Unfortunately, I have no control over the mode of compression, but is there an alternative way of dealing with such a file?  </li>
</ul>