{"action":"created","issue":{"url":"https://api.github.com/repos/KylinOLAP/Kylin/issues/364","labels_url":"https://api.github.com/repos/KylinOLAP/Kylin/issues/364/labels{/name}","comments_url":"https://api.github.com/repos/KylinOLAP/Kylin/issues/364/comments","events_url":"https://api.github.com/repos/KylinOLAP/Kylin/issues/364/events","html_url":"https://github.com/KylinOLAP/Kylin/issues/364","id":53650724,"number":364,"title":"Too high cardinality is not suitable for dictionary!","user":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars.githubusercontent.com/u/1426912?v=3","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-07T16:27:15Z","updated_at":"2015-01-08T01:32:42Z","closed_at":null,"body":"Hi !\r\nWith building a cube faild, it throws some error.\r\n```\r\n[QuartzScheduler_Worker-22]:[2015-01-08 00:21:38,468][INFO][com.kylinolap.dict.DictionaryGenerator.buildDictionaryFromValueList(DictionaryGenerator.java:72)] - Dictionary cardinality 9999956\r\n[QuartzScheduler_Worker-22]:[2015-01-08 00:21:38,468][ERROR][com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:55)] - Too high cardinality is not suitable for dictionary! Are the values stable enough for incremental load??\r\njava.lang.IllegalArgumentException: Too high cardinality is not suitable for dictionary! Are the values stable enough for incremental load??\r\n        at com.kylinolap.dict.DictionaryGenerator.buildDictionaryFromValueList(DictionaryGenerator.java:75)\r\n        at com.kylinolap.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:110)\r\n        at com.kylinolap.dict.DictionaryManager.buildDictionary(DictionaryManager.java:166)\r\n        at com.kylinolap.cube.CubeManager.buildDictionary(CubeManager.java:171)\r\n```\r\nin source code \r\n```\r\n/**\r\n * @author yangli9\r\n */\r\n@SuppressWarnings({ \"rawtypes\", \"unchecked\" })\r\npublic class DictionaryGenerator {\r\n\r\n    private static final Logger logger = LoggerFactory.getLogger(DictionaryGenerator.class);\r\n\r\n    private static final String[] DATE_PATTERNS = new String[] { \"yyyy-MM-dd\" };\r\n\r\n    public static Dictionary<?> buildDictionaryFromValueList(DictionaryInfo info, List<byte[]> values) {\r\n        info.setCardinality(values.size());\r\n...\r\n        // log a few samples\r\n        StringBuilder buf = new StringBuilder();\r\n        for (Object s : samples) {\r\n            if (buf.length() > 0)\r\n                buf.append(\", \");\r\n            buf.append(s.toString()).append(\"=>\").append(dict.getIdFromValue(s));\r\n        }\r\n        logger.info(\"Dictionary value samples: \" + buf.toString());\r\n        logger.info(\"Dictionary cardinality \" + info.getCardinality());\r\n\r\n        if (values.size() > 1000000)\r\n            throw new IllegalArgumentException(\"Too high cardinality is not suitable for dictionary! Are the values stable enough for incremental load??\");\r\n\r\n        return dict;\r\n...\r\n```\r\nHere is limit to 1000000, what is it means?"},"comment":{"url":"https://api.github.com/repos/KylinOLAP/Kylin/issues/comments/69123108","html_url":"https://github.com/KylinOLAP/Kylin/issues/364#issuecomment-69123108","issue_url":"https://api.github.com/repos/KylinOLAP/Kylin/issues/364","id":69123108,"user":{"login":"binmahone","id":6416599,"avatar_url":"https://avatars.githubusercontent.com/u/6416599?v=3","gravatar_id":"","url":"https://api.github.com/users/binmahone","html_url":"https://github.com/binmahone","followers_url":"https://api.github.com/users/binmahone/followers","following_url":"https://api.github.com/users/binmahone/following{/other_user}","gists_url":"https://api.github.com/users/binmahone/gists{/gist_id}","starred_url":"https://api.github.com/users/binmahone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binmahone/subscriptions","organizations_url":"https://api.github.com/users/binmahone/orgs","repos_url":"https://api.github.com/users/binmahone/repos","events_url":"https://api.github.com/users/binmahone/events{/privacy}","received_events_url":"https://api.github.com/users/binmahone/received_events","type":"User","site_admin":false},"created_at":"2015-01-08T01:32:42Z","updated_at":"2015-01-08T01:32:42Z","body":"dictionary resides in memory. if a column has a quite large cardinality, it means the generated dictionary will occupy a lot memory, which does not make a lot sense. For such columns, you might consider avoid using dictionary encoding"}}