{"push_id":745509787,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"687c8c37150f4c93f8e57d86bb56321a4891286b","before":"4cdd8ecd66769316e8593da7790b84cd867968cd","commits":[{"sha":"687c8c37150f4c93f8e57d86bb56321a4891286b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9372] [SQL] Filter nulls in join keys\n\nThis PR adds an optimization rule, `FilterNullsInJoinKey`, to add `Filter` before join operators to filter out rows having null values for join keys.\n\nThis optimization is guarded by a new SQL conf, `spark.sql.advancedOptimization`.\n\nThe code in this PR was authored by yhuai; I'm opening this PR to factor out this change from #7685, a larger pull request which contains two other optimizations.\n\nAuthor: Yin Huai <yhuai@databricks.com>\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #7768 from JoshRosen/filter-nulls-in-join-key and squashes the following commits:\n\nc02fc3f [Yin Huai] Address Josh's comments.\n0a8e096 [Yin Huai] Update comments.\nea7d5a6 [Yin Huai] Make sure we do not keep adding filters.\nbe88760 [Yin Huai] Make it clear that FilterNullsInJoinKeySuite.scala is used to test FilterNullsInJoinKey.\n8bb39ad [Yin Huai] Fix non-deterministic tests.\n303236b [Josh Rosen] Revert changes that are unrelated to null join key filtering\n40eeece [Josh Rosen] Merge remote-tracking branch 'origin/master' into filter-nulls-in-join-key\nc57a954 [Yin Huai] Bug fix.\nd3d2e64 [Yin Huai] First round of cleanup.\nf9516b0 [Yin Huai] Style\nc6667e7 [Yin Huai] Add PartitioningCollection.\ne616d3b [Yin Huai] wip\n7c2d2d8 [Yin Huai] Bug fix and refactoring.\n69bb072 [Yin Huai] Introduce NullSafeHashPartitioning and NullUnsafePartitioning.\nd5b84c3 [Yin Huai] Do not add unnessary filters.\n2201129 [Yin Huai] Filter out rows that will not be joined in equal joins early.","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/687c8c37150f4c93f8e57d86bb56321a4891286b"}]}