{"push_id":809490309,"size":534,"distinct_size":534,"ref":"refs/heads/master","head":"dba95ea03216e6b8e623db4a36e1018c6ed95538","before":"8815ba2f674dbb18eb499216df9942b411e10daa","commits":[{"sha":"864de8eaf4b6ad5c9099f6f29e251c56b029f631","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-9661] [MLLIB] [ML] Java compatibility\n\nI skimmed through the docs for various instance of Object and replaced them with Java compaible versions of the same.\n\n1. Some methods in LDAModel.\n2. runMiniBatchSGD\n3. kolmogorovSmirnovTest\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #8126 from MechCoder/java_incop.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/864de8eaf4b6ad5c9099f6f29e251c56b029f631"},{"sha":"a8d2f4c5f92210a09c846711bd7cc89a43e2edd2","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9942] [PYSPARK] [SQL] ignore exceptions while try to import pandas\n\nIf pandas is broken (can't be imported, raise other exceptions other than ImportError), pyspark can't be imported, we should ignore all the exceptions.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8173 from davies/fix_pandas.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/a8d2f4c5f92210a09c846711bd7cc89a43e2edd2"},{"sha":"c2520f501a200cf794bbe5dc9c385100f518d020","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-9935] [SQL] EqualNotNull not processed in ORC\n\nhttps://issues.apache.org/jira/browse/SPARK-9935\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #8163 from HyukjinKwon/master.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/c2520f501a200cf794bbe5dc9c385100f518d020"},{"sha":"6c5858bc65c8a8602422b46bfa9cf0a1fb296b88","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9922] [ML] rename StringIndexerReverse to IndexToString\n\nWhat `StringIndexerInverse` does is not strictly associated with `StringIndexer`, and the name is not clearly describing the transformation. Renaming to `IndexToString` might be better.\n\n~~I also changed `invert` to `inverse` without arguments. `inputCol` and `outputCol` could be set after.~~\nI also removed `invert`.\n\njkbradley holdenk\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8152 from mengxr/SPARK-9922.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/6c5858bc65c8a8602422b46bfa9cf0a1fb296b88"},{"sha":"693949ba4096c01a0b41da2542ff316823464a16","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-8976] [PYSPARK] fix open mode in python3\n\nThis bug only happen on Python 3 and Windows.\n\nI tested this manually with python 3 and disable python daemon, no unit test yet.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8181 from davies/open_mode.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/693949ba4096c01a0b41da2542ff316823464a16"},{"sha":"c50f97dafd2d5bf5a8351efcc1c8d3e2b87efc72","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9943] [SQL] deserialized UnsafeHashedRelation should be serializable\n\nWhen the free memory in executor goes low, the cached broadcast objects need to serialized into disk, but currently the deserialized UnsafeHashedRelation can't be serialized , fail with NPE. This PR fixes that.\n\ncc rxin\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8174 from davies/serialize_hashed.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/c50f97dafd2d5bf5a8351efcc1c8d3e2b87efc72"},{"sha":"8187b3ae477e2b2987ae9acc5368d57b1d5653b2","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9580] [SQL] Replace singletons in SQL tests\n\nA fundamental limitation of the existing SQL tests is that *there is simply no way to create your own `SparkContext`*. This is a serious limitation because the user may wish to use a different master or config. As a case in point, `BroadcastJoinSuite` is entirely commented out because there is no way to make it pass with the existing infrastructure.\n\nThis patch removes the singletons `TestSQLContext` and `TestData`, and instead introduces a `SharedSQLContext` that starts a context per suite. Unfortunately the singletons were so ingrained in the SQL tests that this patch necessarily needed to touch *all* the SQL test files.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/8111)\n<!-- Reviewable:end -->\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8111 from andrewor14/sql-tests-refactor.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/8187b3ae477e2b2987ae9acc5368d57b1d5653b2"},{"sha":"bd35385d53a6b039e0241e3e73092b8b0a8e455a","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9945] [SQL] pageSize should be calculated from executor.memory\n\nCurrently, pageSize of TungstenSort is calculated from driver.memory, it should use executor.memory instead.\n\nAlso, in the worst case, the safeFactor could be 4 (because of rounding), increase it to 16.\n\ncc rxin\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8175 from davies/page_size.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/bd35385d53a6b039e0241e3e73092b8b0a8e455a"},{"sha":"7c7c7529a16c0e79778e522a3df82a0f1c3762a3","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[MINOR] [SQL] Remove canEqual in Row\n\nAs `InternalRow` does not extend `Row` now, I think we can remove it.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #8170 from viirya/remove_canequal.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/7c7c7529a16c0e79778e522a3df82a0f1c3762a3"},{"sha":"c8677d73666850b37ff937520e538650632ce304","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-9958] [SQL] Make HiveThriftServer2Listener thread-safe and update the tab name to \"JDBC/ODBC Server\"\n\nThis PR fixed the thread-safe issue of HiveThriftServer2Listener, and also changed the tab name to \"JDBC/ODBC Server\" since it's conflict with the new SQL tab.\n\n<img width=\"1377\" alt=\"thriftserver\" src=\"https://cloud.githubusercontent.com/assets/1000778/9265707/c46f3f2c-4269-11e5-8d7e-888c9113ab4f.png\">\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8185 from zsxwing/SPARK-9958.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/c8677d73666850b37ff937520e538650632ce304"},{"sha":"a0e1abbd010b9e73d472ce12ff1d987678005d32","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9661] [MLLIB] minor clean-up of SPARK-9661\n\nSome minor clean-ups after SPARK-9661. See my inline comments. MechCoder jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8190 from mengxr/SPARK-9661-fix.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/a0e1abbd010b9e73d472ce12ff1d987678005d32"},{"sha":"7ecf0c46990c39df8aeddbd64ca33d01824bcc0a","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9956] [ML] Make trees work with one-category features\n\nThis modifies DecisionTreeMetadata construction to treat 1-category features as continuous, so that trees do not fail with such features.  It is important for the pipelines API, where VectorIndexer can automatically categorize certain features as categorical.\n\nAs stated in the JIRA, this is a temp fix which we can improve upon later by automatically filtering out those features. That will take longer, though, since it will require careful indexing.\n\nTargeted for 1.5 and master\n\nCC: manishamde  mengxr yanboliang\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #8187 from jkbradley/tree-1cat.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/7ecf0c46990c39df8aeddbd64ca33d01824bcc0a"},{"sha":"a7317ccdc20d001e5b7f5277b0535923468bfbc6","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-8744] [ML] Add a public constructor to StringIndexer\n\nIt would be helpful to allow users to pass a pre-computed index to create an indexer, rather than always going through StringIndexer to create the model.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #7267 from holdenk/SPARK-8744-StringIndexerModel-should-have-public-constructor.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/a7317ccdc20d001e5b7f5277b0535923468bfbc6"},{"sha":"34d610be854d2a975d9c1e232d87433b85add6fd","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9929] [SQL] support metadata in withColumn\n\nin MLlib sometimes we need to set metadata for the new column, thus we will alias the new column with metadata before call `withColumn` and in `withColumn` we alias this clolumn again. Here I overloaded `withColumn` to allow user set metadata, just like what we did  for `Column.as`.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8159 from cloud-fan/withColumn.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/34d610be854d2a975d9c1e232d87433b85add6fd"},{"sha":"57c2d08800a506614b461b5a505a1dd1a28a8908","author":{"email":"8d2d538dbcb9eb2250781c2df783d51543333c0a@cloudera.com","name":"Neelesh Srinivas Salian"},"message":"[SPARK-9923] [CORE] ShuffleMapStage.numAvailableOutputs should be an Int instead of Long\n\nModified type of ShuffleMapStage.numAvailableOutputs from Long to Int\n\nAuthor: Neelesh Srinivas Salian <nsalian@cloudera.com>\n\nCloses #8183 from nssalian/SPARK-9923.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/57c2d08800a506614b461b5a505a1dd1a28a8908"},{"sha":"3bc55287220b1248e935bf817d880ff176ad4d3b","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9946] [SPARK-9589] [SQL] fix NPE and thread-safety in TaskMemoryManager\n\nCurrently, we access the `page.pageNumer` after it's freed, that could be modified by other thread, cause NPE.\n\nThe same TaskMemoryManager could be used by multiple threads (for example, Python UDF and TransportScript), so it should be thread safe to allocate/free memory/page. The underlying Bitset and HashSet are not thread safe, we should put them inside a synchronized block.\n\ncc JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8177 from davies/memory_manager.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/3bc55287220b1248e935bf817d880ff176ad4d3b"},{"sha":"ece00566e4d5f38585f2810bef38e526cae7d25e","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9561] Re-enable BroadcastJoinSuite\n\nWe can do this now that SPARK-9580 is resolved.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8208 from andrewor14/reenable-sql-tests.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/ece00566e4d5f38585f2810bef38e526cae7d25e"},{"sha":"ffa05c84fe75663fc33f3d954d1cb1e084ab3280","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-9828] [PYSPARK] Mutable values should not be default arguments\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #8110 from MechCoder/spark-9828.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/ffa05c84fe75663fc33f3d954d1cb1e084ab3280"},{"sha":"33bae585d4cb25aed2ac32e0d1248f78cc65318b","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-9809] Task crashes because the internal accumulators are not properly initialized\n\nWhen a stage failed and another stage was resubmitted with only part of partitions to compute, all the tasks failed with error message: java.util.NoSuchElementException: key not found: peakExecutionMemory.\nThis is because the internal accumulators are not properly initialized for this stage while other codes assume the internal accumulators always exist.\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #8090 from carsonwang/SPARK-9809.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/33bae585d4cb25aed2ac32e0d1248f78cc65318b"},{"sha":"6518ef63037aa56b541927f99ad26744f91098ce","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9948] Fix flaky AccumulatorSuite - internal accumulators\n\nIn these tests, we use a custom listener and we assert on fields in the stage / task completion events. However, these events are posted in a separate thread so they're not guaranteed to be posted in time. This commit fixes this flakiness through a job end registration callback.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8176 from andrewor14/fix-accumulator-suite.","distinct":true,"url":"https://api.github.com/repos/jayantshekhar/spark/commits/6518ef63037aa56b541927f99ad26744f91098ce"}]}