{"action":"opened","issue":{"url":"https://api.github.com/repos/SubtitleEdit/subtitleedit/issues/1012","labels_url":"https://api.github.com/repos/SubtitleEdit/subtitleedit/issues/1012/labels{/name}","comments_url":"https://api.github.com/repos/SubtitleEdit/subtitleedit/issues/1012/comments","events_url":"https://api.github.com/repos/SubtitleEdit/subtitleedit/issues/1012/events","html_url":"https://github.com/SubtitleEdit/subtitleedit/issues/1012","id":94883777,"number":1012,"title":"Fix common OCR errors - hardcoded rules - continued from #976","user":{"login":"aaaxx","id":6186480,"avatar_url":"https://avatars.githubusercontent.com/u/6186480?v=3","gravatar_id":"","url":"https://api.github.com/users/aaaxx","html_url":"https://github.com/aaaxx","followers_url":"https://api.github.com/users/aaaxx/followers","following_url":"https://api.github.com/users/aaaxx/following{/other_user}","gists_url":"https://api.github.com/users/aaaxx/gists{/gist_id}","starred_url":"https://api.github.com/users/aaaxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaaxx/subscriptions","organizations_url":"https://api.github.com/users/aaaxx/orgs","repos_url":"https://api.github.com/users/aaaxx/repos","events_url":"https://api.github.com/users/aaaxx/events{/privacy}","received_events_url":"https://api.github.com/users/aaaxx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-14T07:21:24Z","updated_at":"2015-07-14T07:21:24Z","closed_at":null,"body":"Continued from issue https://github.com/SubtitleEdit/subtitleedit/issues/976\r\n\r\n**aaaxx:**\r\n\r\n> In **Fix common errors...**, the function **Fix common OCR errors (using OCR replace list)** marks proper curly apostrophes (’) as errors and replaces them with straight apostrophes (').\r\n\r\n**aaaxx:**\r\n\r\n> I did some some poking around the repository (I'm not a programmer) and I think this is to blame for the curly quotes thing:\r\n\r\n> src/Logic/Dictionaries/OcrFixReplaceList.cs\r\n```csharp\r\nif (word.Contains('’'))\r\n                    word = word.Replace('’', '\\'');\r\n```\r\nDeleting those two lines should fix the problem.\r\n\r\n**ivandrofly:**\r\n\r\n> That feature will be available only if you enable the Hardcoded rule\r\nYou can disable it from Setting/Tools\r\n\r\n**aaaxx:**\r\n\r\n> Thanks, didn't know that. I'd suggest it to be moved to **Tools → Fix common errors...**.\r\n\r\n> However, that `if` condition should still be removed. And this one along with it, because there's no point in dumbing down proper typographic characters:\r\n```csharp\r\nif (word.Contains('—'))\r\n                    word = word.Replace('—', '-'); // em dash to hyphen (my comment)\r\n```\r\nAlso, assuming that `word.Contains` means that the character is immediately surrounded by letters, the replacement character in this next statement should be either spaced en dash (\" – \") or unspaced em dash (\"—\") based on American vs. British English. So maybe it would be a good idea to remove it also because I don't know how you'd decide which one to hardcode, and replacing it with a single hyphen here is not just wrong but will also make it look confusing because you're essentially making two words into a compound word when in reality the second one starts a parenthetical statement inside a sentence. For example:\r\n\"Supposing I had it – which I don’t – do you really think I’d just hand it over to you?\"\r\n\"Supposing I had it-which I don’t-do you really think I’d just hand it over to you?\"\r\n```csharp\r\nwhile (word.Contains(\"--\"))\r\n                    word = word.Replace(\"--\", \"-\");\r\n```\r\n(BTW, I don't know why `while` is used here instead of `if`. Is it something other than a normal replacement?)\r\n\r\n**niksedk:**\r\n\r\n> Tesseract has problems with quotes... so the quick fix was just replacing different quotes with the straight one used in ASCII encoding and the same goes for mdash. Perhaps we should do it more correctly - like \"don't -> don’t \" etc., but lets wait for next update as it is a larger thing to fix... also it looks like Tesseract will (maybe) finally be out in a new version: https://groups.google.com/forum/#!topic/tesseract-dev/8e0F2cK2YzU (no windows binary available yet)\r\n\r\n**ivandrofly:**\r\n\r\n>> BTW, I don't know why while is used here instead of if. Is it something other than a normal replacement?)\r\n\r\n> Its used to fix recursive dash like: -----\r\n\r\n**aaaxx:**\r\n\r\n>> Tesseract has problems with quotes... so the quick fix was just replacing different quotes with the straight one used in ASCII encoding ...\r\n\r\n> OK then, but you should also add acute accent (´) to those replacements, I've seen it happen quite often.\r\n\r\n>> ... and the same goes for mdash.\r\n\r\n> But this one is still very troublesome because when you have a real occurrence of em dash and replace it, you get very confusing results, like that example in my last post. Did you have any real cases where a hyphen was incorrectly recognised as em dash, because I can't remember ever seeing that?\r\n\r\n>> Perhaps we should do it more correctly - like \"don't -> don’t \" etc., but lets wait for next update as it is a larger thing to fix...\r\n\r\n> I don't think that's really necessary. It's quite easy to correctly replace them later. I, for example,  use regex `(?<!\\p{L})['‘’‚]` to check if any of them at word beginnings should be opening single quote, then just replace ASCII apostrophes with curly ones, and then use `(?<=\\d) *[“”„\"‘’‚']` to check if any are foot/inch marks since they should be straight (actually a prime symbol, if you want to split hairs). Besides, I don't know how realistic it is to rely on future contributors to use proper apostrophes when updating word lists.\r\n\r\n\r\n>> ivandrofly:\r\nIts used to fix recursive dash like: -----\r\n\r\n> But wouldn't that replace only occurrences with even number of hyphens since you're looking for *two* hyphens that repeat?\r\n\r\n**niksedk:**\r\n\r\n> @aaaxx: Please create a new issue for this as it's a larger issue - not like changing a font color ;)"}}