{"action":"created","issue":{"url":"https://api.github.com/repos/ryanfb/latinocr-lat/issues/3","labels_url":"https://api.github.com/repos/ryanfb/latinocr-lat/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/ryanfb/latinocr-lat/issues/3/comments","events_url":"https://api.github.com/repos/ryanfb/latinocr-lat/issues/3/events","html_url":"https://github.com/ryanfb/latinocr-lat/issues/3","id":115507607,"number":3,"title":"Improve for scientific Latin","user":{"login":"wollmers","id":1275557,"avatar_url":"https://avatars.githubusercontent.com/u/1275557?v=3","gravatar_id":"","url":"https://api.github.com/users/wollmers","html_url":"https://github.com/wollmers","followers_url":"https://api.github.com/users/wollmers/followers","following_url":"https://api.github.com/users/wollmers/following{/other_user}","gists_url":"https://api.github.com/users/wollmers/gists{/gist_id}","starred_url":"https://api.github.com/users/wollmers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wollmers/subscriptions","organizations_url":"https://api.github.com/users/wollmers/orgs","repos_url":"https://api.github.com/users/wollmers/repos","events_url":"https://api.github.com/users/wollmers/events{/privacy}","received_events_url":"https://api.github.com/users/wollmers/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-11-06T13:45:19Z","updated_at":"2015-11-10T23:08:12Z","closed_at":null,"body":"Did my first tests with scientific Latin texts. While usually with German fraktur the results of tesseract are significantly better than the ABBY results on archive.org, it seems the opposite with Latin.\r\n\r\nExample files:\r\nhttps://github.com/wollmers/ocr-lat-bio-testfiles/tree/master/Scopoli_1763_vindobona\r\n\r\nComparison script:\r\nhttps://github.com/wollmers/ocr-measures/blob/master/ocr_compare.pl\r\n\r\ntesseract https://github.com/wollmers/ocr-lat-bio-testfiles/blob/master/Scopoli_1763_vindobona/ioannisantoiisc03scop_08.tess_3.04lat.compare.txt\r\n\r\n                 lines  words  chars\r\n    items ocr:      20     99    726 \r\n    items grt:      20    106    708 \r\n    matches:         1     43    558 \r\n    edits:          19     67    199 \r\n     subss:         19     52    119 \r\n     inserts:        0      4     49 \r\n     deletions:      0     11     31 \r\n    precision:  0.0500 0.4343 0.7686 \r\n    recall:     0.0500 0.4057 0.7881 \r\n    accuracy:   0.0500 0.3909 0.7371 \r\n    f-score:    0.0500 0.4195 0.7782\r\n\r\nabby https://github.com/wollmers/ocr-lat-bio-testfiles/blob/master/Scopoli_1763_vindobona/ioannisantoniisc03scop_08.abby.compare.txt\r\n\r\n                 lines  words  chars\r\n    items ocr:      20     86    687 \r\n    items grt:      20    106    707 \r\n    matches:         1     48    628 \r\n    edits:          19     58     86 \r\n     subss:         19     38     52 \r\n     inserts:        0      0      7 \r\n     deletions:      0     20     27 \r\n    precision:  0.0500 0.5581 0.9141 \r\n    recall:     0.0500 0.4528 0.8883 \r\n    accuracy:   0.0500 0.4528 0.8796 \r\n    f-score:    0.0500 0.5000 0.9010\r\n\r\nWith this clean test image (no distorts, no shadows, no mud) I would expect scores for words around 0.9 and for chars around 0.95 without special training.\r\n\r\nWhat can be the reasons?\r\n\r\n- missing fonts?\r\n- to many characters supported?\r\n- narrow focus on classic Latin?\r\n- wordlists?\r\n\r\nI can try in the next days to get better results with modified training, expanded wordlist (will keep the new ones extra as I do it with orthographic variants).\r\n\r\nMaybe my lat.traineddata is outdated, but it is the official one fetched from tesseract-ocr.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ryanfb/latinocr-lat/issues/comments/155596776","html_url":"https://github.com/ryanfb/latinocr-lat/issues/3#issuecomment-155596776","issue_url":"https://api.github.com/repos/ryanfb/latinocr-lat/issues/3","id":155596776,"user":{"login":"wollmers","id":1275557,"avatar_url":"https://avatars.githubusercontent.com/u/1275557?v=3","gravatar_id":"","url":"https://api.github.com/users/wollmers","html_url":"https://github.com/wollmers","followers_url":"https://api.github.com/users/wollmers/followers","following_url":"https://api.github.com/users/wollmers/following{/other_user}","gists_url":"https://api.github.com/users/wollmers/gists{/gist_id}","starred_url":"https://api.github.com/users/wollmers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wollmers/subscriptions","organizations_url":"https://api.github.com/users/wollmers/orgs","repos_url":"https://api.github.com/users/wollmers/repos","events_url":"https://api.github.com/users/wollmers/events{/privacy}","received_events_url":"https://api.github.com/users/wollmers/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T23:08:12Z","updated_at":"2015-11-10T23:08:12Z","body":"I don't know where or why it exactly fails.\r\n\r\nThe *tr files are all present.\r\n\r\nWhat I copied from the console is AFAIR the last where it remains for ~10 hrs consuming 100% CPU of one core, up to 16 GB memory:\r\n\r\n    Extracting unicharset from lat.liga.EBGaramondItalic.exp3.box\r\n    Wrote unicharset file ./unicharset.\r\n    set_unicharset_properties -U unicharset -O lat.earlyunicharset --script_dir .\r\n    Loaded unicharset of size 140 from file unicharsetSetting unichar propertiesOther case JOINED of   Joined is not in unicharsetOther case |BROKEN|0|1 of |Broken|0|1 is not in unicharsetOther case ST of ſt is not in unicharsetOther case US of us is not in unicharsetOther case IS of is is not in unicharsetOther case AS of as is not in unicharsetOther case FF of ff is not in unicharsetOther case ES of es is not in unicharsetOther case FI of fi is not in unicharsetOther case FFL of ffl is not in unicharsetOther case FL of fl is not in unicharsetOther case GG of gg is not in unicharsetOther case SB of ſb is not in unicharsetOther case FK of fk is not in unicharsetOther case GY of gy is not in unicharsetOther case FFI of ffi is not in unicharsetOther case FT of ft is not in unicharsetOther case SH of ſh is not in unicharsetOther case SI of ſi is not in unicharsetOther case SS of ſs is not in unicharsetOther case SJ of ſj is not in unicharsetWriting unicharset to file lat.earlyunicharsetrm unicharset\r\n    mftraining -F font_properties -U lat.earlyunicharset -O lat.unicharset lat*tr\r\n    Warning: No shape table file present: shapetable\r\n    Reading lat.EBGaramond.exp-1.tr ...\r\n\r\nUnfortunately I could not see an error message why lat.tessdata was not written, if there was one, because I run it in an SSH-session on my MacAir connected to a remote servers. I should do it more reliable the next time.\r\n\r\nNB: Normalizing the long s to round s does not improve the measures. Seems tessseract has problems with the segmentation and recognition of italic fonts with such fine lines. This m-n-u confusion illustrates this:\r\n\r\n    $ perl -e 'use String::Similarity;print similarity(\"eonnnnnicaio\",\"communicato\"),\"\\n\";'\r\n    0.521739130434783\r\n\r\nThis is nearly impossible to spell correct automagically.\r\n"}}