{"push_id":694500558,"size":4,"distinct_size":4,"ref":"refs/heads/master","head":"d986fb9a378416248768828e6e6c7405697f9a5a","before":"e9471d3414d327c7d0853e18f1844ab1bd09c8ed","commits":[{"sha":"6e9c3ff1ecaf12a0126d83f27f5a4153ae420a34","author":{"email":"f35ee5c3fe1ea39c30a950bdac161ab9d70271b9@netflix.com","name":"Roger Menezes"},"message":"[SPARK-8314][MLlib] improvement in performance of MLUtils.appendBias\n\nMLUtils.appendBias method is heavily used in creating intercepts for linear models.\nThis method uses Breeze's vector concatenation which is very slow compared to the plain\nSystem.arrayCopy. This improvement is to change the implementation to use System.arrayCopy.\n\nI saw the following performance improvements after the change:\nBenchmark with mnist dataset for 50 times:\nMLUtils.appendBias (SparseVector Before): 47320 ms\nMLUtils.appendBias (SparseVector After): 1935 ms\nMLUtils.appendBias (DenseVector Before): 5340 ms\nMLUtils.appendBias (DenseVector After): 4080 ms\nThis is almost a 24 times performance boost for SparseVectors.\n\nAuthor: Roger Menezes <rmenezes@netflix.com>\n\nCloses #6768 from rogermenezes/improve-append-bias and squashes the following commits:\n\n4e42f75 [Roger Menezes] address feedback\ne999d79 [Roger Menezes] first commit","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/6e9c3ff1ecaf12a0126d83f27f5a4153ae420a34"},{"sha":"d46f8e5d4b5c1278e0fae3ad133b2229ac01b197","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-7186] [SQL] Decouple internal Row from external Row\n\nCurrently, we use o.a.s.sql.Row both internally and externally. The external interface is wider than what the internal needs because it is designed to facilitate end-user programming. This design has proven to be very error prone and cumbersome for internal Row implementations.\n\nAs a first step, we create an InternalRow interface in the catalyst module, which is identical to the current Row interface. And we switch all internal operators/expressions to use this InternalRow instead. When we need to expose Row, we convert the InternalRow implementation into Row for users.\n\nFor all public API, we use Row (for example, data source APIs), which will be converted into/from InternalRow by CatalystTypeConverters.\n\nFor all internal data sources (Json, Parquet, JDBC, Hive), we use InternalRow for better performance, casted into Row in buildScan() (without change the public API). When create a PhysicalRDD, we cast them back to InternalRow.\n\ncc rxin marmbrus JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #6792 from davies/internal_row and squashes the following commits:\n\nf2abd13 [Davies Liu] fix scalastyle\na7e025c [Davies Liu] move InternalRow into catalyst\n30db8ba [Davies Liu] Merge branch 'master' of github.com:apache/spark into internal_row\n7cbced8 [Davies Liu] separate Row and InternalRow","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/d46f8e5d4b5c1278e0fae3ad133b2229ac01b197"},{"sha":"4aed66f299a67f5a594da9316b6bf4c345838216","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-8329][SQL] Allow _ in DataSource options\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #6786 from marmbrus/optionsParser and squashes the following commits:\n\ne7d18ef [Michael Armbrust] add dots\n99a3452 [Michael Armbrust] [SPARK-8329][SQL] Allow _ in DataSource options","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/4aed66f299a67f5a594da9316b6bf4c345838216"},{"sha":"d986fb9a378416248768828e6e6c7405697f9a5a","author":{"email":"a6beb8a5d89069108e69bd55bd09a72bc68f527d@measite.de","name":"Rene Treffer"},"message":"[SPARK-7897] Improbe type for jdbc/\"unsigned bigint\"\n\nThe original fix uses DecimalType.Unlimited, which is harder to\nhandle afterwards. There is no scale and most data should fit into\na long, thus DecimalType(20,0) should be better.\n\nAuthor: Rene Treffer <treffer@measite.de>\n\nCloses #6789 from rtreffer/spark-7897-unsigned-bigint-as-decimal and squashes the following commits:\n\n2006613 [Rene Treffer] Fix type for \"unsigned bigint\" jdbc loading.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/d986fb9a378416248768828e6e6c7405697f9a5a"}]}