{"action":"created","issue":{"url":"https://api.github.com/repos/behdad/fonttools/issues/236","labels_url":"https://api.github.com/repos/behdad/fonttools/issues/236/labels{/name}","comments_url":"https://api.github.com/repos/behdad/fonttools/issues/236/comments","events_url":"https://api.github.com/repos/behdad/fonttools/issues/236/events","html_url":"https://github.com/behdad/fonttools/issues/236","id":68841143,"number":236,"title":"[SPEC] Legacy Mac name table entries with PID=1","user":{"login":"twardoch","id":519108,"avatar_url":"https://avatars.githubusercontent.com/u/519108?v=3","gravatar_id":"","url":"https://api.github.com/users/twardoch","html_url":"https://github.com/twardoch","followers_url":"https://api.github.com/users/twardoch/followers","following_url":"https://api.github.com/users/twardoch/following{/other_user}","gists_url":"https://api.github.com/users/twardoch/gists{/gist_id}","starred_url":"https://api.github.com/users/twardoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twardoch/subscriptions","organizations_url":"https://api.github.com/users/twardoch/orgs","repos_url":"https://api.github.com/users/twardoch/repos","events_url":"https://api.github.com/users/twardoch/events{/privacy}","received_events_url":"https://api.github.com/users/twardoch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-04-16T04:21:18Z","updated_at":"2015-04-16T08:55:02Z","closed_at":null,"body":"For the lack for a better place at this point, I'm putting this useful piece of information here as it relates potentially to https://github.com/behdad/fonttools/pull/235 . This info should actually be part of the OpenType or OFF spec, but isn’t. \r\n\r\n# Legacy Mac `name` table entries with `PID=1` #\r\n\r\nAs a rule, the `name` table strings are encoded using Unicode (UTF-16BE). However, the legacy Mac `name` table strings (with `PID=1`) use legacy “Script Manager” Mac encodings, not Unicode. \r\n\r\nTo read them in correctly and write them out correctly into real Unicode, one would either need external libraries such as ICU encodings or Python codecs. ICU brings a set of converters, and the ICU “windows-100x” converters are in fact converters for legacy Mac encodings. Python also brings a set of codecs. But this is not *entirely helpful* because neither ICU nor Python actually bring codecs for *all* of the old Mac encodings that can potentially occur in fonts. Of course it would be stupid to add ICU as a dependency to fontTools, so, while I’m including these tables for people’s reference, I’m not actually advocating that we should convert to and from Unicode for the `PID=1` entries. \r\n\r\n**Instead**, the non-Unicode portions should be encoded using some *escapement*, and ideally somehow denoted as such. This might be *custom escapement* such as `\\xNN` rather than `&#NN`; entities, because we claim that contents of the XML is UTF-8. This is crucial: we should *really* make it clear in tools such as fontTools and in files such as .ttx that a certain string *is Unicode* or that it is *not Unicode*. That aspect of `name` table is not really well solved at all in fontTools/TTX, and I cannot see how any simple patches that we do will make it really good. \r\n\r\n*Note: In the following tables, × means that an encoding is not supported by Python or ICU. and * denotes “emulated” solutions that don’t guarantee full compatibility (see last section).*\r\n\r\n## Latin-script `EID=0` entries ##\r\n\r\nFor `EID=0`, the conversions are also dependent on the `LID`. The following is the mapping from `EID` and `LID` combinations to Python codecs (from the `codecs` module), ICUEncodingIDs, CoreFoundation LIDNames, MIMEnames and WindowsCodepageIDs. The hyperlinks connect to the mapping tables available at ICU and, in the MIMEName column, at unicode.org. \r\n\r\nEID | LID | Python | ICUEncID | LIDName | MIMEName | WinCPID | \r\n--- | --- | ------ | -------- | ------- | -------- | ------- | \r\n0 | 15 | `mac_iceland` | [`macos-37_5-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-37_5-10.2.ucm) | Icelandic | [`x-mac-icelandic`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ICELAND.TXT) | 10079 | \r\n0 | 17 | `mac_turkish` | [`macos-35-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-35-10.2.ucm) | Turkish | [`x-mac-turkish`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/TURKISH.TXT) | 10081 | \r\n0 | 18 | × | [`macos-36_2-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-36_2-10.2.ucm) | Croatian | [`x-mac-croatian`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CROATIAN.TXT) | 10082 | \r\n0 | 24 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Lithuanian | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 25 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Polish | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 26 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Hungarian | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 27 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Estonian | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 28 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Latvian | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 36 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Albanian | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 37 | × | [`macos-38_2-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-38_2-10.2.ucm) | Romanian | [`x-mac-romanian`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMANIAN.TXT) | 10010 | \r\n0 | 38 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Czech | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 39 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Slovak | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | 40 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | Slovenian | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029 | \r\n0 | other | `mac_roman` | [`macos-0_2-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-0_2-10.2.ucm) | Roman | [`x-mac-roman`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT) | 10000 | \r\n\r\n## Non-Latin entries (other `EID` values) ##\r\n\r\nThe following is the mapping from `EID`s to Python encodings, ICUEncodingIDs, CoreFoundation EIDNames, MIMEnames, WindowsCodepageIDs and a fallback info. \r\n\r\nEID | Python | ICUEncID | EIDName | MIMEName | WinCPID\r\n--- | ------ | -------- | ------- | -------- | -------\r\n1 | `shift_jis`* | [`windows-10001-2000`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/windows-10001-2000.ucm) | Japanese | [`x-mac-japanese`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/JAPANESE.TXT) | 10001\r\n2 | `big5`* | [`windows-10002-2000`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/windows-10002-2000.ucm) | ChineseTrad | [`x-mac-chinesetrad`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CHINTRAD.TXT) | 10002\r\n3 | `euc_kr`* | [`windows-10003-2000`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/windows-10003-2000.ucm) | Korean | [`x-mac-korean`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/KOREAN.TXT) | 10003\r\n4 | × | [`windows-10004-2000`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/windows-10004-2000.ucm) | Arabic | [`x-mac-arabic`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ARABIC.TXT) | 10004\r\n5 | × | [`windows-10005-2000`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/windows-10005-2000.ucm) | Hebrew | [`x-mac-hebrew`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/HEBREW.TXT) | 10005\r\n6 | `mac_greek` | [`macos-6_2-10.4`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-6_2-10.4.ucm) | Greek | [`x-mac-greek`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/GREEK.TXT) | 10006\r\n7 | `mac_cyrillic` | [`macos-7_3-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-7_3-10.2.ucm) | Cyrillic | [`x-mac-cyrillic`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CYRILLIC.TXT) | 10007\r\n9 | × | × | Devanagari | [`x-mac-devanagari`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/DEVANAGA.TXT) | ×\r\n10 | × | × | Gurmukhi | [`x-mac-gurmukhi`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/GURMUKHI.TXT) | ×\r\n11 | × | × | Gujarati | [`x-mac-gujarati`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/GUJARATI.TXT) | ×\r\n12 | × | × | Oriya | `x-mac-oriya` | ×\r\n13 | × | × | Bengali | `x-mac-bengali` | ×\r\n14 | × | × | Tamil | `x-mac-tamil` | ×\r\n15 | × | × | Telugu | `x-mac-telugu` | ×\r\n16 | × | × | Kannada | `x-mac-kannada` | ×\r\n17 | × | × | Malayalam | `x-mac-malayalam` | ×\r\n18 | × | × | Sinhalese | `x-mac-sinhalese` | ×\r\n19 | × | × | Burmese | `x-mac-burmese` | ×\r\n20 | × | × | Khmer | `x-mac-khmer` | ×\r\n21 | × | [`macos-21-10.5`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-21-10.5.ucm) | Thai | [`x-mac-thai`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/THAI.TXT) | 10021\r\n22 | × | × | Laotian | `x-mac-lao` | ×\r\n23 | × | × | Georgian | `x-mac-georgian` | ×\r\n24 | × | × | Armenian | `x-mac-armenian` | ×\r\n25 | `gb2312`* | [`windows-10008-2000`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/windows-10008-2000.ucm) | ChineseSimp | [`x-mac-chinesesimp`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CHINSIMP.TXT) | 10008\r\n26 | × | × | Tibetan | `x-mac-tibetan` | ×\r\n27 | × | × | Mongolian | `x-mac-mongolian` | ×\r\n28 | × | × | Ethiopic | `x-mac-ethiopic` | ×\r\n29 | `mac_latin2` | [`macos-29-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-29-10.2.ucm) | CentralEurRoman | [`x-mac-ce`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CENTEURO.TXT) | 10029\r\n30 | × | × | Vietnamese | `x-mac-vietnamese` | ×\r\n31 | × | × | ExtArabic | `x-mac-extarabic` | ×\r\n35 | `mac_turkish` | [`macos-35-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-35-10.2.ucm) | Turkish | [`x-mac-turkish`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/TURKISH.TXT) | 10081\r\n36 | × | [`macos-36_2-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-36_2-10.2.ucm) | Croatian | [`x-mac-croatian`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CROATIAN.TXT) | 10082\r\n37 | `mac_iceland` | [`macos-37_5-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-37_5-10.2.ucm) | Icelandic | [`x-mac-icelandic`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ICELAND.TXT) | 10079\r\n38 | × | [`macos-38_2-10.2`](http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/macos-38_2-10.2.ucm) | Romanian | [`x-mac-romanian`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMANIAN.TXT) | 10010\r\n\r\nIn total, `x-mac-arabic, x-mac-devanagari, x-mac-gurmukhi, x-mac-hebrew, x-mac-thai, x-mac-gujarati, x-mac-croatian, x-mac-romanian` don’t have built-in Python codecs but those could be easily generated from the mapping tables at unicode.org and packaged. \r\n\r\n## Emulated conversion for Asian/CJK ##\r\n\r\nThis is included only for scenarios where one wouldn’t have access to ICU encodings, for example in Python. The Asian legacy Mac encodings are modified and extended versions of some regional Asian “standardized encodings” for the regions: \r\n* **Mac Japanese** is an extended/modified version of **Shift JIS**\r\n* **Mac Traditional Chinese** is an extended/modified version of **Big5**\r\n* **Mac Korean** is an extended/modified version of **EUC-KR**\r\n* **Mac Simplified Chinese** is an extended/modified version of **GB2312**\r\n\r\nSo if one were to accept the risk that a small number of rarely-used characters would not convert fully correctly, converting to/from Unicode using the Python built-in `shift_jis`, `big5`, `euc_kr` and `gb2312` codecs could be used effectively to “emulate” the conversion to/from Unicode of the “real” Mac encodings. It would be minimally-lossy conversion, but in the vast majority of cases it should work fine. \r\n\r\nThis following tables include the byte hex values of the special-handling characters and their corresponding Unicode mappings. The special handling could take into account that the Python codecs used to emulate the Asian legacy Mac encodings do not have a few characters which are present in the legacy Mac encodings (i.e. they’re in the Apple “extensions” of those original encodings), and those characters *can* occur in `name` entries quite frequently. Those characters *are* in the Mac encodings but *are not* in the Python codecs for the “emulation” encodings. \r\n\r\nFor example, check the [`x-mac-korean`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/KOREAN.TXT) mapping table. There is a section there: \r\n```\r\n # Apple additions\r\n 0x81\t0x20A9\t# WON SIGN\r\n 0x83\t0x00A9\t# COPYRIGHT SIGN\r\n```\r\nThese are “Apple additions” which are not present in the `euc_kr` encoding but are in the Apple extensions. (There are many multibyte extensions that Apple added there, but they can be ignored — there is a *very low* probability that they will occur in `name` strings.)\r\n\r\nA possible implementation could be: \r\n* Read the real bytes of the `name` record\r\n* If `EID` is 1, 2, 3 or 25, convert the ByteValue (listed in the tables below) values to some temporary representation (e.g. if you find `\\xFE`, convert it to `&#x2122;`)\r\n* Decode the resulting string using the appropriate “emulation” encoding\r\n* Convert the temporary representation (`&#x2122;`) back into a real UTF-8 character. \r\n\r\nOf course this handling could be embedded in custom `x-mac-japanese-ttx`, `x-mac-chinesetrad-ttx`, `x-mac-korean-ttx` and `x-mac-chinesesimp-ttx` Python codecs that would be “imperfect” (as they wouldn’t support *all* of Apple’s extensions but only the single-byte ones) but still much better than what we have today. **Alternatively** of course, the mapping tables published on unicode.org could be used to generate fully compatible codecs, bypassing the “emulation” and “special handling” trickery. \r\n\r\n### Japanese (`EID=1`)\r\n\r\nWith Japanese (`EID=1`), the Python `shift_jis` codec can emulate the Mac [`x-mac-japanese`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/JAPANESE.TXT) encoding. The `x-mac-japanese-ttx` codec could use `shift_jis` but add special handling as follows: \r\n\r\nByteValue | UniChar | UniHex\r\n--------- | ------- | ------\r\n`\\xFC` | `|` | `\\u007C`\r\n`\\x7E` | `~` | `\\u007E`\r\n`\\x80` | `\\` | `\\u005C`\r\n`\\xA0` | ` ` | `\\u00A0`\r\n`\\xFD` | `©` | `\\u00A9`\r\n`\\xFE` | `™` | `\\u2122`\r\n`\\xFF` | `…` | `\\u2026`\r\n\r\n### Traditional Chinese (`EID=2`)\r\n\r\nWith Traditional Chinese (`EID=2`), the Python `big5` codec can emulate the Mac [`x-mac-chinesetrad`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CHINTRAD.TXT) encoding. The `x-mac-chinesetrad-ttx` codec could use `big5` but add special handling as follows: \r\n\r\nByteValue | UniChar | UniHex\r\n--------- | ------- | ------\r\n`\\x80` | `\\` | `\\u005C`\r\n`\\xA0` | ` ` | `\\u00A0`\r\n`\\xFD` | `©` | `\\u00A9`\r\n`\\xFE` | `™` | `\\u2122`\r\n`\\xFF` | `…` | `\\u2026`\r\n\r\n### Korean (`EID=3`)\r\n\r\nWith Korean (`EID=3`), the Python `euc_kr` codec can emulate the Mac [`x-mac-korean`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/KOREAN.TXT) encoding. The `x-mac-korean-ttx` codec could use `euc_kr` but add special handling as follows: \r\n\r\nByteValue | UniChar | UniHex\r\n--------- | ------- | ------\r\n`\\x80` | ` ` | `\\u00A0`\r\n`\\x81` | `₩` | `\\u20A9`\r\n`\\x82` | `—` | `\\u2014`\r\n`\\x83` | `©` | `\\u00A9`\r\n`\\xFE` | `™` | `\\u2122`\r\n`\\xFF` | `…` | `\\u2026`\r\n\r\n### Simplified Chinese (`EID=25`)\r\n\r\nWith Simplified Chinese (`EID=25`), the Python `gb2312` codec can emulate the Mac [`x-mac-chinesesimp`](http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/CHINSIMP.TXT) encoding. The `x-mac-chinesesimp-ttx` codec could use `gb2312` but add special handling as follows: \r\n\r\nByteValue | UniChar | UniHex\r\n--------- | ------- | ------\r\n`\\x80` | `ü` | `\\u00FC`\r\n`\\xA0` | ` ` | `\\u00A0`\r\n`\\xFD` | `©` | `\\u00A9`\r\n`\\xFE` | `™` | `\\u2122`\r\n`\\xFF` | `…` | `\\u2026`\r\n"},"comment":{"url":"https://api.github.com/repos/behdad/fonttools/issues/comments/93682225","html_url":"https://github.com/behdad/fonttools/issues/236#issuecomment-93682225","issue_url":"https://api.github.com/repos/behdad/fonttools/issues/236","id":93682225,"user":{"login":"twardoch","id":519108,"avatar_url":"https://avatars.githubusercontent.com/u/519108?v=3","gravatar_id":"","url":"https://api.github.com/users/twardoch","html_url":"https://github.com/twardoch","followers_url":"https://api.github.com/users/twardoch/followers","following_url":"https://api.github.com/users/twardoch/following{/other_user}","gists_url":"https://api.github.com/users/twardoch/gists{/gist_id}","starred_url":"https://api.github.com/users/twardoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twardoch/subscriptions","organizations_url":"https://api.github.com/users/twardoch/orgs","repos_url":"https://api.github.com/users/twardoch/repos","events_url":"https://api.github.com/users/twardoch/events{/privacy}","received_events_url":"https://api.github.com/users/twardoch/received_events","type":"User","site_admin":false},"created_at":"2015-04-16T08:55:02Z","updated_at":"2015-04-16T08:55:02Z","body":"CJK are quite important because localized `name` strings are popular with CJK fonts. Also, the legacy CJK encodings are nothing like Unicode, so it’s very confusing to have them left unconverted. But for complex scripts, it doesn’t make sense to have localized Mac legacy names in OpenType fonts. I’ll try to do some usage stats but not soon. In fact, I may run and publish some comprehensive stats on a large font collection. "}}