{"action":"closed","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/10366","labels_url":"https://api.github.com/repos/pydata/pandas/issues/10366/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/10366/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/10366/events","html_url":"https://github.com/pydata/pandas/issues/10366","id":88776630,"number":10366,"title":"Irregular errors when reading certain categorical strings from hdf","user":{"login":"cottrell","id":223276,"avatar_url":"https://avatars.githubusercontent.com/u/223276?v=3","gravatar_id":"","url":"https://api.github.com/users/cottrell","html_url":"https://github.com/cottrell","followers_url":"https://api.github.com/users/cottrell/followers","following_url":"https://api.github.com/users/cottrell/following{/other_user}","gists_url":"https://api.github.com/users/cottrell/gists{/gist_id}","starred_url":"https://api.github.com/users/cottrell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cottrell/subscriptions","organizations_url":"https://api.github.com/users/cottrell/orgs","repos_url":"https://api.github.com/users/cottrell/repos","events_url":"https://api.github.com/users/cottrell/events{/privacy}","received_events_url":"https://api.github.com/users/cottrell/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Bug","name":"Bug","color":"e10c02"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Difficulty%20Intermediate","name":"Difficulty Intermediate","color":"fbca04"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Effort%20Low","name":"Effort Low","color":"006b75"},{"url":"https://api.github.com/repos/pydata/pandas/labels/IO%20HDF5","name":"IO HDF5","color":"5319e7"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Unicode","name":"Unicode","color":"444444"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/34","html_url":"https://github.com/pydata/pandas/milestones/0.17.0","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/34/labels","id":1033710,"number":34,"title":"0.17.0","description":"","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":61,"closed_issues":354,"state":"open","created_at":"2015-03-23T10:47:38Z","updated_at":"2015-08-28T02:29:04Z","due_on":"2015-08-31T04:00:00Z","closed_at":null},"comments":8,"created_at":"2015-06-16T16:33:23Z","updated_at":"2015-08-28T02:29:03Z","closed_at":"2015-08-28T02:29:03Z","body":"It seems that there is something bad happening when we use certain strings with special characters AND the empty string with categoricals:\r\n\r\n```python\r\n    # -*- coding: latin-1 -*-\r\n    import pandas\r\n    import os\r\n\r\n    examples = [\r\n            pandas.Series(['EÉ, 17', '', 'a', 'b', 'c'], dtype='category'),\r\n            pandas.Series(['EÉ, 17', 'a', 'b', 'c'], dtype='category'),\r\n            pandas.Series(['', 'a', 'b', 'c'], dtype='category'),\r\n            pandas.Series(['EE, 17', '', 'a', 'b', 'c'], dtype='category'),\r\n            pandas.Series(['øü', 'a', 'b', 'c'], dtype='category'),\r\n            pandas.Series(['Aøü', '', 'a', 'b', 'c'], dtype='category'),\r\n            pandas.Series(['EÉ, 17', 'øü', 'a', 'b', 'c'], dtype='category')\r\n            ]\r\n\r\n    def test_hdf(s):\r\n        f = 'testhdf.h5'\r\n        if os.path.exists(f):\r\n            os.remove(f)\r\n        s.to_hdf(f, 'data', format='table')\r\n        return pandas.read_hdf(f, 'data')\r\n\r\n    for i, s in enumerate(examples):\r\n        flag = True\r\n        e = ''\r\n        try:\r\n            test_hdf(s)\r\n        except Exception as ex:\r\n            e = ex\r\n            flag = False\r\n        print('%d: %s\\t%s\\t%s' % (i, 'pass' if flag else 'fail', s.tolist(), e))\r\n```\r\nResults in:\r\n\r\n```\r\n    0: fail ['EÉ, 17', '', 'a', 'b', 'c']   Categorical categories must be unique\r\n    1: pass ['EÉ, 17', 'a', 'b', 'c']\r\n    2: pass ['', 'a', 'b', 'c']\r\n    3: pass ['EE, 17', '', 'a', 'b', 'c']\r\n    4: pass ['øü', 'a', 'b', 'c']\r\n    5: fail ['Aøü', '', 'a', 'b', 'c']  Categorical categories must be unique\r\n    6: pass ['EÉ, 17', 'øü', 'a', 'b', 'c']\r\n```\r\n\r\nNot sure if I am using this incorrectly or if this is actually a corner case."}}