using wholeTextFiles in pyspark but get the error of out of memory
<p>I have some files (part-00000.gz, part-00001.gz, part-00002.gz, ...) and each part is rather large. I need to use the filename of each part because it contains time stamp information. As I know, it seems that in pyspark only wholeTextFiles can read input as （filename, content）. However, i get the error of out of memory when using wholeTextFiles. So, my guess is that wholeTextFiles reads a whole part as content in mapper without partition operation. I also find this answer (<a href="https://stackoverflow.com/questions/33906583/how-does-the-number-of-partitions-affect-wholetextfiles-and-textfiles">How does the number of partitions affect `wholeTextFiles` and `textFiles`?</a>). If so, how can i get the filename of a rather large part file. Thanks</p>