{"push_id":731834269,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"a1064df0ee3daf496800be84293345a10e1497d9","before":"dac7dbf5a6bd663ef3acaa5b2249b31140fa6857","commits":[{"sha":"a1064df0ee3daf496800be84293345a10e1497d9","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-8125] [SQL] Accelerates Parquet schema merging and partition discovery\n\nThis PR tries to accelerate Parquet schema discovery and `HadoopFsRelation` partition discovery.  The acceleration is done by the following means:\n\n- Turning off schema merging by default\n\n  Schema merging is not the most common case, but requires reading footers of all Parquet part-files and can be very slow.\n\n- Avoiding `FileSystem.globStatus()` call when possible\n\n  `FileSystem.globStatus()` may issue multiple synchronous RPC calls, and can be very slow (esp. on S3).  This PR adds `SparkHadoopUtil.globPathIfNecessary()`, which only issues RPC calls when the path contain glob-pattern specific character(s) (`{}[]*?\\`).\n\n  This is especially useful when converting a metastore Parquet table with lots of partitions, since Spark SQL adds all partition directories as the input paths, and currently we do a `globStatus` call on each input path sequentially.\n\n- Listing leaf files in parallel when the number of input paths exceeds a threshold\n\n  Listing leaf files is required by partition discovery.  Currently it is done on driver side, and can be slow when there are lots of (nested) directories, since each `FileSystem.listStatus()` call issues an RPC.  In this PR, we list leaf files in a BFS style, and resort to a Spark job once we found that the number of directories need to be listed exceed a threshold.\n\n  The threshold is controlled by `SQLConf` option `spark.sql.sources.parallelPartitionDiscovery.threshold`, which defaults to 32.\n\n- Discovering Parquet schema in parallel\n\n  Currently, schema merging is also done on driver side, and needs to read footers of all part-files.  This PR uses a Spark job to do schema merging.  Together with task side metadata reading in Parquet 1.7.0, we never read any footers on driver side now.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7396 from liancheng/accel-parquet and squashes the following commits:\n\n5598efc [Cheng Lian] Uses ParquetInputFormat[InternalRow] instead of ParquetInputFormat[Row]\nff32cd0 [Cheng Lian] Excludes directories while listing leaf files\n3c580f1 [Cheng Lian] Fixes test failure caused by making \"mergeSchema\" default to \"false\"\nb1646aa [Cheng Lian] Should allow empty input paths\n32e5f0d [Cheng Lian] Moves schema merging to executor side","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/a1064df0ee3daf496800be84293345a10e1497d9"}]}