{"action":"created","issue":{"url":"https://api.github.com/repos/neovim/neovim/issues/2159","labels_url":"https://api.github.com/repos/neovim/neovim/issues/2159/labels{/name}","comments_url":"https://api.github.com/repos/neovim/neovim/issues/2159/comments","events_url":"https://api.github.com/repos/neovim/neovim/issues/2159/events","html_url":"https://github.com/neovim/neovim/pull/2159","id":61869500,"number":2159,"title":"[RFC] Generate unicode tables with script when building","user":{"login":"ZyX-I","id":322097,"avatar_url":"https://avatars.githubusercontent.com/u/322097?v=3","gravatar_id":"","url":"https://api.github.com/users/ZyX-I","html_url":"https://github.com/ZyX-I","followers_url":"https://api.github.com/users/ZyX-I/followers","following_url":"https://api.github.com/users/ZyX-I/following{/other_user}","gists_url":"https://api.github.com/users/ZyX-I/gists{/gist_id}","starred_url":"https://api.github.com/users/ZyX-I/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ZyX-I/subscriptions","organizations_url":"https://api.github.com/users/ZyX-I/orgs","repos_url":"https://api.github.com/users/ZyX-I/repos","events_url":"https://api.github.com/users/ZyX-I/events{/privacy}","received_events_url":"https://api.github.com/users/ZyX-I/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/neovim/neovim/labels/RFC","name":"RFC","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-03-15T17:15:56Z","updated_at":"2015-03-15T18:09:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/neovim/neovim/pulls/2159","html_url":"https://github.com/neovim/neovim/pull/2159","diff_url":"https://github.com/neovim/neovim/pull/2159.diff","patch_url":"https://github.com/neovim/neovim/pull/2159.patch"},"body":"This commits removes large tables from mbyte.c and adds a script that generates them. Added lua script is a translation of unicode.vim tool (removed from the repository) to lua.\r\n\r\nDownloading of file is done using `file(DOWNLOAD` when running CMake to generate build files and I am not entirely sure this is what should be done.\r\n\r\nThis results in the following update to the unicode tables:\r\n\r\n```Patch\r\n--- /tmp/unicode_tables.old.generated.h\t2015-03-15 20:13:21.438407881 +0300\r\n+++ build/src/nvim/auto/unicode_tables.generated.h\t2015-03-15 20:13:01.848017764 +0300\r\n@@ -65,2 +65,3 @@ static const convertStruct toLower[] = {\r\n   {0x376, 0x376, -1, 1},\r\n+  {0x37f, 0x37f, -1, 116},\r\n   {0x386, 0x386, -1, 38},\r\n@@ -84,5 +85,6 @@ static const convertStruct toLower[] = {\r\n   {0x4c1, 0x4cd, 2, 1},\r\n-  {0x4d0, 0x524, 2, 1},\r\n+  {0x4d0, 0x52e, 2, 1},\r\n   {0x531, 0x556, 1, 48},\r\n   {0x10a0, 0x10c5, 1, 7264},\r\n+  {0x10c7, 0x10cd, 6, 7264},\r\n   {0x1e00, 0x1e94, 2, 1},\r\n@@ -134,5 +136,5 @@ static const convertStruct toLower[] = {\r\n   {0x2ceb, 0x2ced, 2, 1},\r\n-  {0xa640, 0xa65e, 2, 1},\r\n-  {0xa662, 0xa66c, 2, 1},\r\n-  {0xa680, 0xa696, 2, 1},\r\n+  {0x2cf2, 0xa640, 31054, 1},\r\n+  {0xa642, 0xa66c, 2, 1},\r\n+  {0xa680, 0xa69a, 2, 1},\r\n   {0xa722, 0xa72e, 2, 1},\r\n@@ -143,4 +145,14 @@ static const convertStruct toLower[] = {\r\n   {0xa78b, 0xa78b, -1, 1},\r\n+  {0xa78d, 0xa78d, -1, -42280},\r\n+  {0xa790, 0xa792, 2, 1},\r\n+  {0xa796, 0xa7a8, 2, 1},\r\n+  {0xa7aa, 0xa7aa, -1, -42308},\r\n+  {0xa7ab, 0xa7ab, -1, -42319},\r\n+  {0xa7ac, 0xa7ac, -1, -42315},\r\n+  {0xa7ad, 0xa7ad, -1, -42305},\r\n+  {0xa7b0, 0xa7b0, -1, -42258},\r\n+  {0xa7b1, 0xa7b1, -1, -42282},\r\n   {0xff21, 0xff3a, 1, 32},\r\n   {0x10400, 0x10427, 1, 40},\r\n+  {0x118a0, 0x118bf, 1, 32},\r\n };\r\n@@ -199,4 +211,8 @@ static const convertStruct toUpper[] = {\r\n   {0x25b, 0x25b, -1, -203},\r\n+  {0x25c, 0x25c, -1, 42319},\r\n   {0x260, 0x260, -1, -205},\r\n+  {0x261, 0x261, -1, 42315},\r\n   {0x263, 0x263, -1, -207},\r\n+  {0x265, 0x265, -1, 42280},\r\n+  {0x266, 0x266, -1, 42308},\r\n   {0x268, 0x268, -1, -209},\r\n@@ -204,2 +220,3 @@ static const convertStruct toUpper[] = {\r\n   {0x26b, 0x26b, -1, 10743},\r\n+  {0x26c, 0x26c, -1, 42305},\r\n   {0x26f, 0x26f, -1, -211},\r\n@@ -210,2 +227,3 @@ static const convertStruct toUpper[] = {\r\n   {0x280, 0x283, 3, -218},\r\n+  {0x287, 0x287, -1, 42282},\r\n   {0x288, 0x288, -1, -218},\r\n@@ -215,2 +233,3 @@ static const convertStruct toUpper[] = {\r\n   {0x292, 0x292, -1, -219},\r\n+  {0x29e, 0x29e, -1, 42258},\r\n   {0x345, 0x345, -1, 84},\r\n@@ -235,2 +254,3 @@ static const convertStruct toUpper[] = {\r\n   {0x3f2, 0x3f2, -1, 7},\r\n+  {0x3f3, 0x3f3, -1, -116},\r\n   {0x3f5, 0x3f5, -1, -96},\r\n@@ -243,3 +263,3 @@ static const convertStruct toUpper[] = {\r\n   {0x4cf, 0x4cf, -1, -15},\r\n-  {0x4d1, 0x525, 2, -1},\r\n+  {0x4d1, 0x52f, 2, -1},\r\n   {0x561, 0x586, 1, -48},\r\n@@ -286,6 +306,7 @@ static const convertStruct toUpper[] = {\r\n   {0x2cec, 0x2cee, 2, -1},\r\n+  {0x2cf3, 0x2cf3, -1, -1},\r\n   {0x2d00, 0x2d25, 1, -7264},\r\n-  {0xa641, 0xa65f, 2, -1},\r\n-  {0xa663, 0xa66d, 2, -1},\r\n-  {0xa681, 0xa697, 2, -1},\r\n+  {0x2d27, 0x2d2d, 6, -7264},\r\n+  {0xa641, 0xa66d, 2, -1},\r\n+  {0xa681, 0xa69b, 2, -1},\r\n   {0xa723, 0xa72f, 2, -1},\r\n@@ -294,5 +315,8 @@ static const convertStruct toUpper[] = {\r\n   {0xa77f, 0xa787, 2, -1},\r\n-  {0xa78c, 0xa78c, -1, -1},\r\n+  {0xa78c, 0xa791, 5, -1},\r\n+  {0xa793, 0xa797, 4, -1},\r\n+  {0xa799, 0xa7a9, 2, -1},\r\n   {0xff41, 0xff5a, 1, -32},\r\n   {0x10428, 0x1044f, 1, -40},\r\n+  {0x118c0, 0x118df, 1, -32},\r\n };\r\n@@ -307,6 +331,6 @@ static const struct interval combining[]\r\n   {0x0610, 0x061a},\r\n-  {0x064b, 0x065e},\r\n+  {0x064b, 0x065f},\r\n   {0x0670, 0x0670},\r\n   {0x06d6, 0x06dc},\r\n-  {0x06de, 0x06e4},\r\n+  {0x06df, 0x06e4},\r\n   {0x06e7, 0x06e8},\r\n@@ -321,6 +345,7 @@ static const struct interval combining[]\r\n   {0x0829, 0x082d},\r\n-  {0x0900, 0x0903},\r\n-  {0x093c, 0x093c},\r\n-  {0x093e, 0x094e},\r\n-  {0x0951, 0x0955},\r\n+  {0x0859, 0x085b},\r\n+  {0x08e4, 0x0903},\r\n+  {0x093a, 0x093c},\r\n+  {0x093e, 0x094f},\r\n+  {0x0951, 0x0957},\r\n   {0x0962, 0x0963},\r\n@@ -359,3 +384,3 @@ static const struct interval combining[]\r\n   {0x0bd7, 0x0bd7},\r\n-  {0x0c01, 0x0c03},\r\n+  {0x0c00, 0x0c03},\r\n   {0x0c3e, 0x0c44},\r\n@@ -365,3 +390,3 @@ static const struct interval combining[]\r\n   {0x0c62, 0x0c63},\r\n-  {0x0c82, 0x0c83},\r\n+  {0x0c81, 0x0c83},\r\n   {0x0cbc, 0x0cbc},\r\n@@ -372,3 +397,3 @@ static const struct interval combining[]\r\n   {0x0ce2, 0x0ce3},\r\n-  {0x0d02, 0x0d03},\r\n+  {0x0d01, 0x0d03},\r\n   {0x0d3e, 0x0d44},\r\n@@ -398,3 +423,3 @@ static const struct interval combining[]\r\n   {0x0f86, 0x0f87},\r\n-  {0x0f90, 0x0f97},\r\n+  {0x0f8d, 0x0f97},\r\n   {0x0f99, 0x0fbc},\r\n@@ -410,3 +435,3 @@ static const struct interval combining[]\r\n   {0x109a, 0x109d},\r\n-  {0x135f, 0x135f},\r\n+  {0x135d, 0x135f},\r\n   {0x1712, 0x1714},\r\n@@ -415,3 +440,3 @@ static const struct interval combining[]\r\n   {0x1772, 0x1773},\r\n-  {0x17b6, 0x17d3},\r\n+  {0x17b4, 0x17d3},\r\n   {0x17dd, 0x17dd},\r\n@@ -427,2 +452,3 @@ static const struct interval combining[]\r\n   {0x1a7f, 0x1a7f},\r\n+  {0x1ab0, 0x1abe},\r\n   {0x1b00, 0x1b04},\r\n@@ -431,3 +457,4 @@ static const struct interval combining[]\r\n   {0x1b80, 0x1b82},\r\n-  {0x1ba1, 0x1baa},\r\n+  {0x1ba1, 0x1bad},\r\n+  {0x1be6, 0x1bf3},\r\n   {0x1c24, 0x1c37},\r\n@@ -436,7 +463,9 @@ static const struct interval combining[]\r\n   {0x1ced, 0x1ced},\r\n-  {0x1cf2, 0x1cf2},\r\n-  {0x1dc0, 0x1de6},\r\n-  {0x1dfd, 0x1dff},\r\n+  {0x1cf2, 0x1cf4},\r\n+  {0x1cf8, 0x1cf9},\r\n+  {0x1dc0, 0x1df5},\r\n+  {0x1dfc, 0x1dff},\r\n   {0x20d0, 0x20f0},\r\n   {0x2cef, 0x2cf1},\r\n+  {0x2d7f, 0x2d7f},\r\n   {0x2de0, 0x2dff},\r\n@@ -445,3 +474,4 @@ static const struct interval combining[]\r\n   {0xa66f, 0xa672},\r\n-  {0xa67c, 0xa67d},\r\n+  {0xa674, 0xa67d},\r\n+  {0xa69f, 0xa69f},\r\n   {0xa6f0, 0xa6f1},\r\n@@ -458,2 +488,3 @@ static const struct interval combining[]\r\n   {0xa9b3, 0xa9c0},\r\n+  {0xa9e5, 0xa9e5},\r\n   {0xaa29, 0xaa36},\r\n@@ -461,3 +492,3 @@ static const struct interval combining[]\r\n   {0xaa4c, 0xaa4d},\r\n-  {0xaa7b, 0xaa7b},\r\n+  {0xaa7b, 0xaa7d},\r\n   {0xaab0, 0xaab0},\r\n@@ -467,2 +498,4 @@ static const struct interval combining[]\r\n   {0xaac1, 0xaac1},\r\n+  {0xaaeb, 0xaaef},\r\n+  {0xaaf5, 0xaaf6},\r\n   {0xabe3, 0xabea},\r\n@@ -471,4 +504,6 @@ static const struct interval combining[]\r\n   {0xfe00, 0xfe0f},\r\n-  {0xfe20, 0xfe26},\r\n+  {0xfe20, 0xfe2d},\r\n   {0x101fd, 0x101fd},\r\n+  {0x102e0, 0x102e0},\r\n+  {0x10376, 0x1037a},\r\n   {0x10a01, 0x10a03},\r\n@@ -478,4 +513,33 @@ static const struct interval combining[]\r\n   {0x10a3f, 0x10a3f},\r\n-  {0x11080, 0x11082},\r\n+  {0x10ae5, 0x10ae6},\r\n+  {0x11000, 0x11002},\r\n+  {0x11038, 0x11046},\r\n+  {0x1107f, 0x11082},\r\n   {0x110b0, 0x110ba},\r\n+  {0x11100, 0x11102},\r\n+  {0x11127, 0x11134},\r\n+  {0x11173, 0x11173},\r\n+  {0x11180, 0x11182},\r\n+  {0x111b3, 0x111c0},\r\n+  {0x1122c, 0x11237},\r\n+  {0x112df, 0x112ea},\r\n+  {0x11301, 0x11303},\r\n+  {0x1133c, 0x1133c},\r\n+  {0x1133e, 0x11344},\r\n+  {0x11347, 0x11348},\r\n+  {0x1134b, 0x1134d},\r\n+  {0x11357, 0x11357},\r\n+  {0x11362, 0x11363},\r\n+  {0x11366, 0x1136c},\r\n+  {0x11370, 0x11374},\r\n+  {0x114b0, 0x114c3},\r\n+  {0x115af, 0x115b5},\r\n+  {0x115b8, 0x115c0},\r\n+  {0x11630, 0x11640},\r\n+  {0x116ab, 0x116b7},\r\n+  {0x16af0, 0x16af4},\r\n+  {0x16b30, 0x16b36},\r\n+  {0x16f51, 0x16f7e},\r\n+  {0x16f8f, 0x16f92},\r\n+  {0x1bc9d, 0x1bc9e},\r\n   {0x1d165, 0x1d169},\r\n@@ -486,2 +550,3 @@ static const struct interval combining[]\r\n   {0x1d242, 0x1d244},\r\n+  {0x1e8d0, 0x1e8d6},\r\n   {0xe0100, 0xe01ef},\r\n@@ -555,2 +620,3 @@ static const convertStruct foldCase[] =\r\n   {0x376, 0x376, -1, 1},\r\n+  {0x37f, 0x37f, -1, 116},\r\n   {0x386, 0x386, -1, 38},\r\n@@ -582,5 +648,6 @@ static const convertStruct foldCase[] =\r\n   {0x4c1, 0x4cd, 2, 1},\r\n-  {0x4d0, 0x524, 2, 1},\r\n+  {0x4d0, 0x52e, 2, 1},\r\n   {0x531, 0x556, 1, 48},\r\n   {0x10a0, 0x10c5, 1, 7264},\r\n+  {0x10c7, 0x10cd, 6, 7264},\r\n   {0x1e00, 0x1e94, 2, 1},\r\n@@ -634,5 +701,5 @@ static const convertStruct foldCase[] =\r\n   {0x2ceb, 0x2ced, 2, 1},\r\n-  {0xa640, 0xa65e, 2, 1},\r\n-  {0xa662, 0xa66c, 2, 1},\r\n-  {0xa680, 0xa696, 2, 1},\r\n+  {0x2cf2, 0xa640, 31054, 1},\r\n+  {0xa642, 0xa66c, 2, 1},\r\n+  {0xa680, 0xa69a, 2, 1},\r\n   {0xa722, 0xa72e, 2, 1},\r\n@@ -643,4 +710,14 @@ static const convertStruct foldCase[] =\r\n   {0xa78b, 0xa78b, -1, 1},\r\n+  {0xa78d, 0xa78d, -1, -42280},\r\n+  {0xa790, 0xa792, 2, 1},\r\n+  {0xa796, 0xa7a8, 2, 1},\r\n+  {0xa7aa, 0xa7aa, -1, -42308},\r\n+  {0xa7ab, 0xa7ab, -1, -42319},\r\n+  {0xa7ac, 0xa7ac, -1, -42315},\r\n+  {0xa7ad, 0xa7ad, -1, -42305},\r\n+  {0xa7b0, 0xa7b0, -1, -42258},\r\n+  {0xa7b1, 0xa7b1, -1, -42282},\r\n   {0xff21, 0xff3a, 1, 32},\r\n   {0x10400, 0x10427, 1, 40},\r\n+  {0x118a0, 0x118bf, 1, 32},\r\n };\r\n@@ -648,4 +725,2 @@ static const struct interval doublewidth\r\n   {0x1100, 0x115f},\r\n-  {0x11a3, 0x11a7},\r\n-  {0x11fa, 0x11ff},\r\n   {0x2329, 0x232a},\r\n@@ -655,9 +730,8 @@ static const struct interval doublewidth\r\n   {0x2ff0, 0x2ffb},\r\n-  {0x3000, 0x3029},\r\n-  {0x3030, 0x303e},\r\n+  {0x3000, 0x303e},\r\n   {0x3041, 0x3096},\r\n-  {0x309b, 0x30ff},\r\n+  {0x3099, 0x30ff},\r\n   {0x3105, 0x312d},\r\n   {0x3131, 0x318e},\r\n-  {0x3190, 0x31b7},\r\n+  {0x3190, 0x31ba},\r\n   {0x31c0, 0x31e3},\r\n@@ -671,4 +745,2 @@ static const struct interval doublewidth\r\n   {0xac00, 0xd7a3},\r\n-  {0xd7b0, 0xd7c6},\r\n-  {0xd7cb, 0xd7fb},\r\n   {0xf900, 0xfaff},\r\n@@ -680,5 +752,7 @@ static const struct interval doublewidth\r\n   {0xffe0, 0xffe6},\r\n-  {0x1f200, 0x1f200},\r\n-  {0x1f210, 0x1f231},\r\n+  {0x1b000, 0x1b001},\r\n+  {0x1f200, 0x1f202},\r\n+  {0x1f210, 0x1f23a},\r\n   {0x1f240, 0x1f248},\r\n+  {0x1f250, 0x1f251},\r\n   {0x20000, 0x2fffd},\r\n@@ -740,2 +814,3 @@ static const struct interval ambiguous[]\r\n   {0x02df, 0x02df},\r\n+  {0x0300, 0x036f},\r\n   {0x0391, 0x03a1},\r\n@@ -850,2 +925,3 @@ static const struct interval ambiguous[]\r\n   {0xe000, 0xf8ff},\r\n+  {0xfe00, 0xfe0f},\r\n   {0xfffd, 0xfffd},\r\n@@ -853,15 +929,5 @@ static const struct interval ambiguous[]\r\n   {0x1f110, 0x1f12d},\r\n-  {0x1f131, 0x1f131},\r\n-  {0x1f13d, 0x1f13d},\r\n-  {0x1f13f, 0x1f13f},\r\n-  {0x1f142, 0x1f142},\r\n-  {0x1f146, 0x1f146},\r\n-  {0x1f14a, 0x1f14e},\r\n-  {0x1f157, 0x1f157},\r\n-  {0x1f15f, 0x1f15f},\r\n-  {0x1f179, 0x1f179},\r\n-  {0x1f17b, 0x1f17c},\r\n-  {0x1f17f, 0x1f17f},\r\n-  {0x1f18a, 0x1f18d},\r\n-  {0x1f190, 0x1f190},\r\n+  {0x1f130, 0x1f169},\r\n+  {0x1f170, 0x1f19a},\r\n+  {0xe0100, 0xe01ef},\r\n   {0xf0000, 0xffffd},\r\n```"},"comment":{"url":"https://api.github.com/repos/neovim/neovim/issues/comments/81181625","html_url":"https://github.com/neovim/neovim/pull/2159#issuecomment-81181625","issue_url":"https://api.github.com/repos/neovim/neovim/issues/2159","id":81181625,"user":{"login":"ZyX-I","id":322097,"avatar_url":"https://avatars.githubusercontent.com/u/322097?v=3","gravatar_id":"","url":"https://api.github.com/users/ZyX-I","html_url":"https://github.com/ZyX-I","followers_url":"https://api.github.com/users/ZyX-I/followers","following_url":"https://api.github.com/users/ZyX-I/following{/other_user}","gists_url":"https://api.github.com/users/ZyX-I/gists{/gist_id}","starred_url":"https://api.github.com/users/ZyX-I/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ZyX-I/subscriptions","organizations_url":"https://api.github.com/users/ZyX-I/orgs","repos_url":"https://api.github.com/users/ZyX-I/repos","events_url":"https://api.github.com/users/ZyX-I/events{/privacy}","received_events_url":"https://api.github.com/users/ZyX-I/received_events","type":"User","site_admin":false},"created_at":"2015-03-15T18:09:28Z","updated_at":"2015-03-15T18:09:28Z","body":"This is variant 2. I am personally OK with this:\r\n\r\n1. Download, put and commit those files into `/unicode` directory.\r\n2. Add a script which does 1. for you.\r\n3. Remove `file(DOWNLOAD` from CMake."}}