{"push_id":543025887,"size":5,"distinct_size":5,"ref":"refs/heads/maint-5.20","head":"ff0890fe397087cb96ed5cab0e5a62eaced63858","before":"48ef8337fed3e33960323ad60ce1b5d3396c56e8","commits":[{"sha":"b58dfac208eabad2bba6ad7680218c4fee116e16","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"PATCH: [perl #123539] regcomp.c node overrun/segfault\n\nThis is a minimal patch suitable for a maintenance release.  It extracts\nthe guts of reguni and REGC without the conditional they have.  The next\ncommit will do some refactoring to avoid branching and to make things\nclearer.\n\nThis bug is due to the current two pass structure of the Perl regular\nexpression compiler.  The first pass attempts to do just enough work to\nfigure out how much space to malloc for the compiled pattern; and the\n2nd pass actually fills in the details.  One problem with this design is\nthat in many cases quite a bit of work is required to figure out the\nsize, and this work is thrown away and redone in the second pass.\nAnother problem is that it is easy to forget to do enough work in the\nsizing pass, and that is what happened with the blamed commit.  I\nunderstand that there are plans (God speed) to change the compiler\ndesign.\n\nWhen not under /i matching, the size of a node that will match a\nsequence of characters is just the number of bytes those characters take\nup.  We have an easy way to calculate the number of bytes any code point\nwill occupy in UTF-8, and it's just 1 byte per code point for non-UTF-8.\nSo in the sizing pass, we don't actually have to figure out the\nrepresentation of the characters.  However under /i matching, we do.\nFirst of all, matching of UTF-8 strings is done by replacing each\ncharacter of each string by its fold-case (function fc()) and then\ncomparing.  This is required by the nature of full Unicode matching\nwhich is not 1-1.  If we do that replacement for the pattern at compile\ntime, we avoid having to do it over-and-over as pattern matching\nbacktracks at execution.  And because fc(x) may not occupy the same\nnumber of bytes as x, and there is no easy way to know that size without\nactually doing the fc(), we have to do the fold in the sizing pass.\nNow, there are relatively few folds where sizeof(fc(x)) != sizeof(x), so\nwe could construct an exception table for those few cases where it is,\nand look up through that.\n\nBut there is another reason that we have to fold in the sizing pass.\nAnd that is because of the potential for multi-character folds being\nsplit across regnodes.  The regular expression compiler generates\nEXACTish regnodes for matching sequences of characters exactly or via\n/i.  The limit for how many bytes in a sequence such a node can match is\n255 because the length is stored in a U8.  If the pattern has a sequence\nlonger than that, it is split into two or more EXACTish nodes in a row.\n(Actually, the compiler splits at a size much lower than that; I'm not\nsure why, but then two adjoining nodes whose total sum length is at most\n255 get joined later in the third, optimizing pass.)  Now consider,\nmatching the character U+FB03 LATIN SMALL LIGATURE FFI.  It matches the\nsequence of the three characters \"f f i\".  Because of the design of the\nregex pattern matching code, if these characters are such that the first\none or two are at the end of one EXACTish node, and the final two or one\nare in another EXACTish node, then U+FB03 wrongly would not match them.\nMatches can't cross node boundaries.  If the pattern were tweaked so all\nthree characters were in either the first or second node, then the match\nwould succeed.  And that is what the compiler does.  When it reaches the\nnode's size limit, and the final character is one that is a non-terminal\ncharacter in a multi-char fold, what's in the node is backed-off until\nit ends with a character without this characteristic.  This has to be\ndone in the sizing pass, as we are repacking the nodes, which can affect\nthe size of the pattern, and we have to know what the folds are in order\nto determine all this.\n\n(We don't fold non-UTF-8 patterns.  This is for two reasons.  One is\nthat one character, the U+00B5 MICRO SIGN, folds to above-Latin1, and if\nwe folded it, we would have to change the pattern into UTF-8, and that\nwould slow everything down.  I've thought about adding a regnode type\nfor the much more common case of a sequence that doesn't have this\ncharacter in it, and which could hence be folded at compile time.  But\nI've not been able to justify this because of the 2nd reason, which is\nfolds in this range are simple enough to be handled by an array lookup,\nso folding is fast at runtime.)\n\nThen there is the complication of matching under locale rules.  This bug\nmanifested itself only under /l matching.  We can't fold at pattern\ncompile time, because the folding rules won't be known until runtime.\nThis isn't a problem for non-UTF-8 locales, as all folds are 1-1, and so\nthere never will be a multi-char fold.  But there could be such folds in\na UTF-8 locale, so the regnodes have to be packed to work for that\neventuality.  The blamed commit did not do that, and because this issue\ndoesn't arise unless there is a string long enough to trigger the\nproblem, this wasn't found until now.  What is needed, and what this\ncommit does, is for the unfolded characters to be accumulated in both\npasses.  The code that looks for potential multi-char fold issues\nhandles both folded and unfolded-inputs, so will work.\n\n(cherry picked from commit 405dffcb17b9cc9d0e5d7b41835b998ca7f1d873)","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/b58dfac208eabad2bba6ad7680218c4fee116e16"},{"sha":"6ecf4c9b3d3ceee25646d9ff9cce633e5a8acde1","author":{"email":"3fc74fd0baa7fa2229210c398a130951d3e32d32@cpan.org","name":"Father Chrysostomos"},"message":"[perl #123410] sort CORE::fake bizarre behaviour\n\nThis commit:\n\ncommit 01b5ef509f2ebf466fd7de2c1e7406717bb14332\nAuthor: Father Chrysostomos <sprout@cpan.org>\nDate:   Fri Jun 7 20:16:23 2013 -0700\n\n    [perl #24482] Fix sort and require to treat CORE:: as keyword\n\ncaused sort CORE::lc \"FOO\" to be equivalent to sort +CORE::lc \"FOO\",\nthe way it does if a keyword is not preceded by CORE::.  But it made\nthe mistake of chopping off the last six characters if it is not a\nkeyword after CORE::.\n\nSo\n\n    sort CORE::f @_\n\nbecame equivalent to\n\n    sort C @_\n\n!\n\nThis commit just reverts to the previous behaviour in such cases.\n\n(cherry picked from commit 487e470dbd7a885bb6a92a735b2783e1c6740066)","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/6ecf4c9b3d3ceee25646d9ff9cce633e5a8acde1"},{"sha":"c53c18b37582ea0d3309c034dc67c4c337838b53","author":{"email":"3fc74fd0baa7fa2229210c398a130951d3e32d32@cpan.org","name":"Father Chrysostomos"},"message":"perldelta for 487e470d / #123410\n\n(cherry picked from commit 070733dfda1bb0d07d337128db5d1e78f14e6562)","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/c53c18b37582ea0d3309c034dc67c4c337838b53"},{"sha":"b8271aef8d2cc621e8de9b6a5c90d4c203b1a4ae","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"perldiag: Add missing entry\n\n(cherry picked from commit faad849dd7af72f2db0a94098b4c462d3e573f5f)","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/b8271aef8d2cc621e8de9b6a5c90d4c203b1a4ae"},{"sha":"ff0890fe397087cb96ed5cab0e5a62eaced63858","author":{"email":"4ea5a6fcb5b7546937a0a105350d8778134c6858@googlemail.com","name":"Steve Hay"},"message":"Fix typo in previous commit\n\nManually backported from 605eee6061b9fb79ab7be45ac13eaf417fd8db4f.","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/ff0890fe397087cb96ed5cab0e5a62eaced63858"}]}