{"push_id":789490280,"size":5,"distinct_size":5,"ref":"refs/heads/blead","head":"512c0f5a351dd399dbb069e222741582044f88cc","before":"6b5cf123f371e012d9812b37b13d50c6e06bf555","commits":[{"sha":"0f0eec8fa0e9c6558ae31d368ff8b2c87314deee","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"regcomp.c: Fix, clarify comments","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/0f0eec8fa0e9c6558ae31d368ff8b2c87314deee"},{"sha":"82a6ada44d6e80d9ab64fe6a0334f04487506fdb","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"regcomp.c: Add assertion and parameter to macro\n\nIt's clearer and safer to pass the name of a local variable to a macro,\nrather than assuming the macro knows the correct name.","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/82a6ada44d6e80d9ab64fe6a0334f04487506fdb"},{"sha":"539601d8d15042c7077d7fecb1acd97058ddd274","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"regcomp.c: Reorder a test\n\nPrior to this commit, the code tested for some side effects before\ntesting if the called function even succeeded.  This hasn't been a\nproblem before, because the called function didn't fail when called\nfrom this context.  But a future commit will change that.","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/539601d8d15042c7077d7fecb1acd97058ddd274"},{"sha":"b97943f7f63f0d1fe0d9501503c553accdbdbb81","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"regcomp.c: Split an internal flag into 2\n\nThis splits the flag used to communicate between parsing layers that the\nsizing pass needs to be restarted and the pattern upgraded to UTF-8.  It\nis split into a bit meaning to restart pass1 and a bit to do the\nupgrade.  This is in preparation for the next commit which will have a\n2nd reason to restart pass1.","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/b97943f7f63f0d1fe0d9501503c553accdbdbb81"},{"sha":"512c0f5a351dd399dbb069e222741582044f88cc","author":{"email":"b58d09249b3e367ae8383be7fc79efabb7794458@cpan.org","name":"Karl Williamson"},"message":"PATCH: [perl #125990] panic: reg_node overrun\n\nThis is a result of a design flaw that I introduced in earlier releases\nwhen attempting to fix earlier design flaws in dealing with the outlier\ncharacter ß, LATIN SMALL LETTER SHARP S.  The uppercase of this letter\nis SS, so that when comparing case-insensitively, it should match 'ss',\nand hence, in Unicode terminology, it folds to 'ss'.  This character is\nthe only one representable without using UTF-8 whose fold is longer than\n1 byte, and so has to have special treatment.  Similarly, the sequence\n'ss' can match caselessly the single byte ß, and this is the only such\nsequence that can match something shorter than it, unless UTF-8 is\ninvolved.  The matter is complicated by the fact that under /di rules,\nthe ß and 'ss' don't match each other, unless the target string is in\nUTF-8.  The solution I used earlier (and continue to use) was to create\na special regnode EXACTFU_SS under /ui rules, in which any ß is folded\nto 'ss'.  But under /di rules, a regular EXACTF regnode is used, and any\nß is retained as-is.\n\nThe problem reported here arises when something during the sizing pass\ntells perl to use /ui rules rather than the /di rules that were in\neffect at the beginning.  Recall that perl uses /d rules, for backward\ncompatibility, unless something overrides them.  This can be a 'use'\ndeclaration, an explicit character set pattern modifier, or something in\nthe pattern.  This bug happens only with the final case.  There are\nseveral Unicode-defined constructs that can occur in patterns; if one is\nfound, the perl interpreter infers that Unicode is desired, and switches\nfrom /d to /u for the whole pattern.  Two such constructs are a Unicode\nproperty, \\p{}, and a Unicode named character, \\N{}.  The\nproblem-reproducing code for this ticket uses the latter.\n\nThe problem was that the switch from /di to /ui was deferred until AFTER\nthe sizing pass.  (A flag was set when one of these constructs was\nencountered to tell the parser to later do the switch.)\n\nDuring the second pass, the code realizes it is under /ui, so creates an\nEXACTFU_SS node and folds the ß into 'ss'.  But the first pass thought\nit was under /di, so it sized for just the ß, i.e., for 1 byte, so we\nexceed the allocated space and do a wild write.  This may not cause a\nproblem if the malloc'd space had rounded-up and there were only a few\nof these ß characters.\n\nOne solution I considered was just keeping a global count of the ß\ncharacters in EXACTF nodes.  One could just add these to the space\nreserved if /ui rules ended up being used.  The problem with this is that\nnodes that are near their maximum size without the extra space could\nexceed it with, and thus have to be split into 2 nodes, and the extra\nnode would have an unplanned-for header, taking up more unaccounted-for\nspace.  So that doesn't work.  One could also just reserve two bytes for\nevery ß in an EXACTF node, thus wasting space unless /ui ends up being\nused.  But the bigger problem is that the code that splits nodes would\nhave to be made more complicated.  It has to find a suitable splitting\nspot, by searching through the text of the node, and now it would have\nto deal with some of that space not being set.\n\nInstead, I opted to change the code so that when it finds one of these\nUnicode-defined constructs, it switches to /u immediately during the\nsizing pass.  That means that the parse afterwards knows that it is /u\nand allocates the correct space.  (We now have to remain in /u for the\nremainder of the pass, so some code had to change that reverted this.)\nThis fixes the test case in the ticket.  But there remains a problem if\nthe sizing has happened earlier in the parse before the construct that\nchanges from /d to /u is encountered.  Like:\n\n    qr/.....ß....\\N{}/di\n\nThe incorrect sizing has already happened by the time the \\N{} is\nencountered.  One could solve this by restarting the parse whenever the\n/d goes to /u (under /i, as this issue isn't a problem except when\nfolding ß).  That slows things down.  Instead, I opted to set a global\nflag whenever a ß is found in an EXACTF node.  If that flag isn't set at\nthe time of the /d to /u switch, there's no need to restart the parse.\n\nA 'use utf8' or 'use 5.012' or higher selects /u over /d, so the problem\ndid not happen with them, nor if the pattern has to be converted to\nUTF-8, which restarts the sizing pass, and it only happens with the\nsharp s character.  And probably unless there a several ß characters,\nthe rounding-up of malloc space, would cause this to not be an issue.\nThese explain why this hasn't been reported from the field.","distinct":true,"url":"https://api.github.com/repos/Perl/perl5/commits/512c0f5a351dd399dbb069e222741582044f88cc"}]}