{"action":"created","issue":{"url":"https://api.github.com/repos/antlr/antlr4/issues/795","labels_url":"https://api.github.com/repos/antlr/antlr4/issues/795/labels{/name}","comments_url":"https://api.github.com/repos/antlr/antlr4/issues/795/comments","events_url":"https://api.github.com/repos/antlr/antlr4/issues/795/events","html_url":"https://github.com/antlr/antlr4/issues/795","id":54787820,"number":795,"title":"Result of recoverInline is ignored, leading to excess token consumption during recovery","user":{"login":"michaelpj","id":1699466,"avatar_url":"https://avatars.githubusercontent.com/u/1699466?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelpj","html_url":"https://github.com/michaelpj","followers_url":"https://api.github.com/users/michaelpj/followers","following_url":"https://api.github.com/users/michaelpj/following{/other_user}","gists_url":"https://api.github.com/users/michaelpj/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelpj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelpj/subscriptions","organizations_url":"https://api.github.com/users/michaelpj/orgs","repos_url":"https://api.github.com/users/michaelpj/repos","events_url":"https://api.github.com/users/michaelpj/events{/privacy}","received_events_url":"https://api.github.com/users/michaelpj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-01-19T16:59:42Z","updated_at":"2015-01-19T17:13:07Z","closed_at":null,"body":"The method `recoverInline` in `ANTLRErrorStrategy` returns a `Token`, however, in my generated parsers, I see it used like this:\r\n```java\r\n\t\ttry {\r\n\t\t\tenterOuterAlt(_localctx, 1);\r\n\t\t\t{\r\n\t\t\tsetState(479);\r\n\t\t\t_la = _input.LA(1);\r\n\t\t\tif ( !(_la==TKminus || _la==TKplus) ) {\r\n\t\t\t_errHandler.recoverInline(this);\r\n\t\t\t}\r\n\t\t\tconsume();\r\n\t\t\t}\r\n\t\t}\r\n```\r\nThat is, the `Token` produced from `recoverInline` is ignored. \r\n\r\nThis can lead to excess token consumption, as the `DefaultErrorStrategy` will call `consume` if it deletes a token, returning the matched token. (It calls it first in `singleTokenDeletion` to delete the token, but then again in `recoverInline`). But then the parser will `consume` once again, so we also consume the token *after* the one we wanted.\r\n\r\nNeedless to say, this can really mess up error recovery.\r\n\r\nIt's not clear to me from the documentation of `recoverInline` whether a successful return indicates that the token has been consumed or not. I suspect it does, since `singleTokenInsertion` does not in fact insert a token into the stream that could later be consumed. Hence, the fix is probably just to put the generated `consume` in an `else` block."},"comment":{"url":"https://api.github.com/repos/antlr/antlr4/issues/comments/70528383","html_url":"https://github.com/antlr/antlr4/issues/795#issuecomment-70528383","issue_url":"https://api.github.com/repos/antlr/antlr4/issues/795","id":70528383,"user":{"login":"sharwell","id":1408396,"avatar_url":"https://avatars.githubusercontent.com/u/1408396?v=3","gravatar_id":"","url":"https://api.github.com/users/sharwell","html_url":"https://github.com/sharwell","followers_url":"https://api.github.com/users/sharwell/followers","following_url":"https://api.github.com/users/sharwell/following{/other_user}","gists_url":"https://api.github.com/users/sharwell/gists{/gist_id}","starred_url":"https://api.github.com/users/sharwell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sharwell/subscriptions","organizations_url":"https://api.github.com/users/sharwell/orgs","repos_url":"https://api.github.com/users/sharwell/repos","events_url":"https://api.github.com/users/sharwell/events{/privacy}","received_events_url":"https://api.github.com/users/sharwell/received_events","type":"User","site_admin":false},"created_at":"2015-01-19T17:13:07Z","updated_at":"2015-01-19T17:13:07Z","body":"The result of `recoverInline` is assigned to the label variable if your token in the grammar is labeled.\r\n\r\nThere are three different behaviors the default implementation of `recoverInline` gives. Suppose you are trying to match a token `Y` followed by `Z`.\r\n\r\n1. If the input sequence is LA(1)=`X`, LA(2)=`Y`, (i.e. skipping `X` will allow the parser to match `Y`), then we call `consume` to skip `X` and then return `LT(1)`. This code path should work fine. When the call returns, LA(1)=`Y` which means we *should* call `consume` (which it does).\r\n2. If the input sequence is LA(1)=`Z`, then assume the `Y` is missing, and we construct a new token `Y`. When the call returns, LA(1)=`Z` which means we *should not* call `consume`. **This code appears to have a bug.**\r\n3. If neither of the above work, then throw an `InputMismatchException`. This code path should work fine as well."}}