{"action":"created","issue":{"url":"https://api.github.com/repos/earwig/mwparserfromhell/issues/96","labels_url":"https://api.github.com/repos/earwig/mwparserfromhell/issues/96/labels{/name}","comments_url":"https://api.github.com/repos/earwig/mwparserfromhell/issues/96/comments","events_url":"https://api.github.com/repos/earwig/mwparserfromhell/issues/96/events","html_url":"https://github.com/earwig/mwparserfromhell/issues/96","id":54410715,"number":96,"title":"pywikibot pure python template param parser differences","user":{"login":"jayvdb","id":15092,"avatar_url":"https://avatars.githubusercontent.com/u/15092?v=3","gravatar_id":"","url":"https://api.github.com/users/jayvdb","html_url":"https://github.com/jayvdb","followers_url":"https://api.github.com/users/jayvdb/followers","following_url":"https://api.github.com/users/jayvdb/following{/other_user}","gists_url":"https://api.github.com/users/jayvdb/gists{/gist_id}","starred_url":"https://api.github.com/users/jayvdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jayvdb/subscriptions","organizations_url":"https://api.github.com/users/jayvdb/orgs","repos_url":"https://api.github.com/users/jayvdb/repos","events_url":"https://api.github.com/users/jayvdb/events{/privacy}","received_events_url":"https://api.github.com/users/jayvdb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-15T04:12:48Z","updated_at":"2015-01-15T05:13:17Z","closed_at":null,"body":"pywikibot has a pure python template param extractor using regex, which behaves slightly different in corner cases to mwpfh.\r\n\r\npywikibot's mwpfh code is simply using \"parsed_wikitext_obj.filter_templates(recursive=True)\":\r\nhttps://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/textlib.py#L1013\r\n\r\nand the lovely regex solution is directly below that in the same file - lines 1021-1183.\r\n\r\nTests have been written to identify where they behave the same and where they behave differently.  The relevant section starts here:\r\n\r\nhttps://github.com/wikimedia/pywikibot-core/blob/master/tests/textlib_tests.py#L199\r\n\r\n_extract_templates_params() is all the cases which textlib pure python and mwpfh are identical.\r\n\r\ntest_extract_templates_params_mwpfh and test_extract_templates_params_regex each include 10 assertions where they differ - the inputs to each are identical; the output is different.\r\n\r\nThe main difference is that mwpfh doesnt trim whitespace around param names and values.  I'd appreciate knowing whether mwpfh might be interested in adding a mode/flag/whatever which trim's whitespace.\r\n\r\nIf our two projects can agree on expected behaviour, pywikibot can delete its pure python template parser, related tests can be put into mwpfh, and pywikibot adds mwpfh as a mandatory dependency."},"comment":{"url":"https://api.github.com/repos/earwig/mwparserfromhell/issues/comments/70040756","html_url":"https://github.com/earwig/mwparserfromhell/issues/96#issuecomment-70040756","issue_url":"https://api.github.com/repos/earwig/mwparserfromhell/issues/96","id":70040756,"user":{"login":"ricordisamoa","id":4588496,"avatar_url":"https://avatars.githubusercontent.com/u/4588496?v=3","gravatar_id":"","url":"https://api.github.com/users/ricordisamoa","html_url":"https://github.com/ricordisamoa","followers_url":"https://api.github.com/users/ricordisamoa/followers","following_url":"https://api.github.com/users/ricordisamoa/following{/other_user}","gists_url":"https://api.github.com/users/ricordisamoa/gists{/gist_id}","starred_url":"https://api.github.com/users/ricordisamoa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ricordisamoa/subscriptions","organizations_url":"https://api.github.com/users/ricordisamoa/orgs","repos_url":"https://api.github.com/users/ricordisamoa/repos","events_url":"https://api.github.com/users/ricordisamoa/events{/privacy}","received_events_url":"https://api.github.com/users/ricordisamoa/received_events","type":"User","site_admin":false},"created_at":"2015-01-15T05:13:17Z","updated_at":"2015-01-15T05:13:17Z","body":"mwpfh causes almost no loss of data, while pywikibot's regex-based solution does:\r\n* whitespace is stripped, while it could be meaningful to Scribunto modules;\r\n* duplicate arguments (useful, e.g., for https://gerrit.wikimedia.org/r/184543/) cannot be supported due to the use of OrderedDict;\r\n* the parser is unreliable for nested templates, as opposed to mwpfh's robust Wikicode class;\r\n* it even [lost the order of arguments](//phabricator.wikimedia.org/T57882)!.\r\n\r\nIt should be dropped ASAP."}}