{"push_id":616421438,"size":10,"distinct_size":10,"ref":"refs/heads/master","head":"418b8c0c41e8291622bd6c3eeac06e69c0217429","before":"514061c414080701cb046171041a2d00859660e8","commits":[{"sha":"d328af5981322f6741da3ebcd01307259ae326fe","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MEDIUM: compression: postpone buffer adjustments after compression\n\nTill now we used to copy the pending outgoing data into the new buffer,\nthen compute the chunk size, then compress, then fix the chunk size,\nthen copy the remaining data into the destination buffer. If the\ncompression would fail for whatever reason (eg: not enough input bytes\nto push an extra block), this work still had to be performed for no\nadded value. It also presents the disadvantage of having to use a fixed\nlength to encode the chunk size.\n\nThanks to the body parser changes that went late into 1.5, the buffers\nare not modified anymore during these operations. So this patch rearranges\noperations so that they're more optimal :\n\n1) init() prepares a new buffer and reserves space in it for pending\n   outgoing data (no copy) and for chunk size\n2) data are compressed\n3) only if data were added to the buffer, then the old data are copied\n   and the chunk size is set.\n\nA few optimisations are still possible to go further :\n\n  - decide whether we prefer to copy pending outgoing data from the\n    old buffer to the new one, or pending incoming compressed data\n    from the new one to the old one, based on the amount of outgoing\n    data available. Given that pending outgoing data are rare and the\n    operation could be complex in the presence of extra input data,\n    it's probably better to ignore this one ;\n\n  - compute the needed length for the chunk size. This would avoid\n    sending lots of leading zeroes when not needed.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/d328af5981322f6741da3ebcd01307259ae326fe"},{"sha":"15530d28a4ff58a370b27bde9672aaf7dbb7341f","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MEDIUM: compression: don't send leading zeroes with chunk size\n\nTill now we used to rely on a fixed maximum chunk size. Thanks to last\ncommit we're now free to adjust the chunk's length before sending the\ndata, so we don't have to use 6 digits all the time anymore, and if\none wants buffers larger than 16 MB it is now possible.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/15530d28a4ff58a370b27bde9672aaf7dbb7341f"},{"sha":"2aee2215c908c6997addcd1714b5b10f73c0703d","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"BUG/MINOR: compression: consider the expansion factor in init\n\nWhen checking if the buffer is large enough, we used to rely on a fixed\nsize that was \"apparently\" enough. We need to consider the expansion\nfactor of deflate-encoded streams instead, which is of 5 bytes per 32kB.\nThe previous value was OK till 128kB buffers but became wrong past that.\nIt's totally harmless since we always keep the reserve when compressiong,\nso there's 1kB or so available, which is enough for buffers as large as\n6.5 MB, but better fix the check anyway.\n\nThis fix could be backported into 1.5 since compression was added there.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/2aee2215c908c6997addcd1714b5b10f73c0703d"},{"sha":"e7e49a8d0b87e345b9a3648c1aeaf651a0bed1f6","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MINOR: http: check the algo name \"identity\" instead of the function pointer\n\nNext patch will statity all compression functions, so let's stop relying\non a function pointer comparison and use the algo name instead.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/e7e49a8d0b87e345b9a3648c1aeaf651a0bed1f6"},{"sha":"9f640a1eab4bc5ec4ed89a21e670b7ed4388d664","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"CLEANUP: compression: statify all algo-specific functions\n\nThere's no reason for exporting identity_* nor deflate_*, they're only\nused in the same file. Mark them static, it will make it easier to add\nother algorithms.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/9f640a1eab4bc5ec4ed89a21e670b7ed4388d664"},{"sha":"615105e7e88843c43f052a1e2866eb31ccc67cb3","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MEDIUM: compression: add a distinction between UA- and config- algorithms\n\nThanks to MSIE/IIS, the \"deflate\" name is ambigous. According to the RFC\nit's a zlib-wrapped deflate stream, but IIS used to send only a raw deflate\nstream, which is the only format MSIE understands for \"deflate\". The other\nwidely used browsers do support both formats. For this reason some people\nprefer to emit a raw deflate stream on \"deflate\" to serve more users even\nit that means violating the standards. Haproxy only follows the standard,\nso they cannot do this.\n\nThis patch makes it possible to have one algorithm name in the configuration\nand another one in the protocol. This will make it possible to have a new\nconfiguration token to add a different algorithm so that users can decide if\nthey want a raw deflate or the standard one.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/615105e7e88843c43f052a1e2866eb31ccc67cb3"},{"sha":"c91840aa3374548f07029eb3f985f3016ce7f713","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MEDIUM: compression: add new \"raw-deflate\" compression algorithm\n\nThis algorithm is exactly the same as \"deflate\" without the zlib wrapper,\nand used as an alternative when the browser wants \"deflate\". All major\nbrowsers understand it and despite violating the standards, it is known\nto work better than \"deflate\", at least on MSIE and some versions of\nSafari. Do not use it in conjunction with \"deflate\", use either one or\nthe other since both react to the same Accept-Encoding token. Note that\nthe lack of Adler32 checksum makes it slightly faster.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/c91840aa3374548f07029eb3f985f3016ce7f713"},{"sha":"9787efa97c351352ef4250dd5698e9709be1f5e7","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MEDIUM: compression: split deflate_flush() into flush and finish\n\nThis function used to take a zlib-specific flag as argument to indicate\nwhether a buffer flush or end of contents was met, let's split it in two\nso that we don't depend on zlib anymore.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/9787efa97c351352ef4250dd5698e9709be1f5e7"},{"sha":"7b218778886c4e7385330f4b14889675297108f3","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"CLEANUP: compression: remove unused reset functions\n\nIt's unclear what purpose these functions used to server, however they\nare not used anywhere, one good reason to remove them.","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/7b218778886c4e7385330f4b14889675297108f3"},{"sha":"418b8c0c41e8291622bd6c3eeac06e69c0217429","author":{"email":"aff024fe4ab0fece4091de044c58c9ae4233383a@1wt.eu","name":"Willy Tarreau"},"message":"MAJOR: compression: integrate support for libslz\n\nThis library is designed to emit a zlib-compatible stream with no\nmemory usage and to favor resource savings over compression ratio.\nWhile zlib requires 256 kB of RAM per compression context (and can only\nsupport 4000 connections per GB of RAM), the stateless compression\noffered by libslz does not need to retain buffers between subsequent\ncalls. In theory this slightly reduces the compression ratio but in\npractice it does not have that much of an effect since the zlib\nwindow is limited to 32kB.\n\nLibslz is available at :\n\n      http://git.1wt.eu/web?p=libslz.git\n\nIt was designed for web compression and provides a lot of savings\nover zlib in haproxy. Here are the preliminary results on a single\ncore of a core2-quad 3.0 GHz in 32-bit for only 300 concurrent\nsessions visiting the home page of www.haproxy.org (76 kB) with\nthe default 16kB buffers :\n\n          BW In      BW Out     BW Saved   Ratio   memory VSZ/RSS\nzlib      237 Mbps    92 Mbps   145 Mbps   2.58     84M /  69M\nslz       733 Mbps   380 Mbps   353 Mbps   1.93    5.9M / 4.2M\n\nSo while the compression ratio is lower, the bandwidth savings are\nmuch more important due to the significantly lower compression cost\nwhich allows to consume even more data from the servers. In the\nexample above, zlib became the bottleneck at 24% of the output\nbandwidth. Also the difference in memory usage is obvious.\n\nMore tests run on a single core of a core i5-3320M, with 500 concurrent\nusers and the default 16kB buffers :\n\nAt 100% CPU (no limit) :\n          BW In      BW Out     BW Saved   Ratio   memory VSZ/RSS  hits/s\nzlib      480 Mbps   188 Mbps   292 Mbps   2.55     130M / 101M     744\nslz      1700 Mbps   810 Mbps   890 Mbps   2.10    23.7M / 9.7M    2382\n\nAt 85% CPU (limited) :\n          BW In      BW Out     BW Saved   Ratio   memory VSZ/RSS  hits/s\nzlib     1240 Mbps   976 Mbps   264 Mbps   1.27     130M / 100M    1738\nslz      1600 Mbps   976 Mbps   624 Mbps   1.64    23.7M / 9.7M    2210\n\nThe most important benefit really happens when the CPU usage is\nlimited by \"maxcompcpuusage\" or the BW limited by \"maxcomprate\" :\nin order to preserve resources, haproxy throttles the compression\nratio until usage is within limits. Since slz is much cheaper, the\naverage compression ratio is much higher and the input bandwidth\nis quite higher for one Gbps output.\n\nOther tests made with some reference files :\n\n                           BW In     BW Out    BW Saved  Ratio  hits/s\ndaniels.html       zlib  1320 Mbps  163 Mbps  1157 Mbps   8.10    1925\n                   slz   3600 Mbps  580 Mbps  3020 Mbps   6.20    5300\n\ntv.com/listing     zlib   980 Mbps  124 Mbps   856 Mbps   7.90     310\n                   slz   3300 Mbps  553 Mbps  2747 Mbps   5.97    1100\n\njquery.min.js      zlib   430 Mbps  180 Mbps   250 Mbps   2.39     547\n                   slz   1470 Mbps  764 Mbps   706 Mbps   1.92    1815\n\nbootstrap.min.css  zlib   790 Mbps  165 Mbps   625 Mbps   4.79     777\n                   slz   2450 Mbps  650 Mbps  1800 Mbps   3.77    2400\n\nSo on top of saving a lot of memory, slz is constantly 2.5-3.5 times\nfaster than zlib and results in providing more savings for a fixed CPU\nusage. For links smaller than 100 Mbps, zlib still provides a better\ncompression ratio, at the expense of a much higher CPU usage.\n\nLarger input files provide slightly higher bandwidth for both libs, at\nthe expense of a bit more memory usage for zlib (it converges to 256kB\nper connection).","distinct":true,"url":"https://api.github.com/repos/yuxans/haproxy/commits/418b8c0c41e8291622bd6c3eeac06e69c0217429"}]}