{"push_id":848188010,"size":3,"distinct_size":3,"ref":"refs/heads/master","head":"04d1509229d74b3c0b2c207482ec6bd40ca91156","before":"3912eefe4599a790f49b217dc108389515a172e2","commits":[{"sha":"77b392334338c4904aa133696bf34799b2c6954f","author":{"email":"7a9c1c7ad0f46b2149632c353354dd206862d72b@us.ibm.com","name":"Matthias Boehm"},"message":"Improved parallel writers (fallback to sequential for single file)\n\nAll parallel writers in cp determine their degree of parallelism based\non available parallelism and datasize divided by hdfs blocksize. For\ngood scalability, every thread writes it own file (which is consistent\nwith write from distributed backends). However, it leads to\nunnecessarily complex outputs (e.g., \"./A.csv/m-0-00000\") for small data\n(<128MB). This change lets the parallel writers fall back to sequential\nwrite in these situations, which gives a slight benefit in terms of\nperformance but more importantly collapses the unnecessary complex\noutput directories to \"A.csv\".","distinct":true,"url":"https://api.github.com/repos/SparkTC/systemml/commits/77b392334338c4904aa133696bf34799b2c6954f"},{"sha":"44553037236ee756d810d381809e7b4e9b35ee03","author":{"email":"7a9c1c7ad0f46b2149632c353354dd206862d72b@us.ibm.com","name":"Matthias Boehm"},"message":"Improved compilation time (prevent unnecessary cluster status report)","distinct":true,"url":"https://api.github.com/repos/SparkTC/systemml/commits/44553037236ee756d810d381809e7b4e9b35ee03"},{"sha":"04d1509229d74b3c0b2c207482ec6bd40ca91156","author":{"email":"7a9c1c7ad0f46b2149632c353354dd206862d72b@us.ibm.com","name":"Matthias Boehm"},"message":"New simplification rewrite 'fuse leftindexing chain to append', for glm\n\nExample: Y_prob[,1]=A; Y_prob[,2]=B --> Y_prob = cbind(A,B);\nThis rewrite prevents the unnecessary creation of a dense intermediate\nand thus improves performance due to less write and potential buffer\npool evictions. \n\nThis change also includes a cleanup wrt recompilation of append, which\nwas unnecessarily marked always for recompilation. Given correct size\ninference over loops and complex control structures, this special case\nis not required.","distinct":true,"url":"https://api.github.com/repos/SparkTC/systemml/commits/04d1509229d74b3c0b2c207482ec6bd40ca91156"}]}