{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/10180","labels_url":"https://api.github.com/repos/pydata/pandas/issues/10180/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/10180/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/10180/events","html_url":"https://github.com/pydata/pandas/issues/10180","id":78623198,"number":10180,"title":"Segfault when writing data out of order to pd.HDFStore via append","user":{"login":"bastianl","id":8620425,"avatar_url":"https://avatars.githubusercontent.com/u/8620425?v=3","gravatar_id":"","url":"https://api.github.com/users/bastianl","html_url":"https://github.com/bastianl","followers_url":"https://api.github.com/users/bastianl/followers","following_url":"https://api.github.com/users/bastianl/following{/other_user}","gists_url":"https://api.github.com/users/bastianl/gists{/gist_id}","starred_url":"https://api.github.com/users/bastianl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bastianl/subscriptions","organizations_url":"https://api.github.com/users/bastianl/orgs","repos_url":"https://api.github.com/users/bastianl/repos","events_url":"https://api.github.com/users/bastianl/events{/privacy}","received_events_url":"https://api.github.com/users/bastianl/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/HDF5","name":"HDF5","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-05-20T17:47:02Z","updated_at":"2015-05-20T21:58:28Z","closed_at":null,"body":"I am trying to append chunks of data to an (initially empty) HDF5 frame with pd.HDFStore. The chunks come in out of order, and sometimes certain orders produce segfaults. This script seems to consistently segfault after loading file 31 (*update, see comments for better example*). You will notice that by looking at the time stamps outputted by the script it appears to be when hdf5 tries to fill some gap data. I can produce more files that trigger segfaults if necessary.\r\n\r\nI've managed to narrow it down to the following line in `pytables.py`\r\n```\r\n> /home/user/env/lib/python3.4/site-packages/pandas/io/pytables.py(3738)write_data_chunk()\r\n   3737                 self.table.append(rows)\r\n-> 3738                 self.table.flush()\r\n   3739         except Exception as detail:\r\n```\r\n```\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.4.2.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 3.16.0-4-amd64\r\nmachine: x86_64\r\nprocessor:\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_US.UTF-8\r\n\r\npandas: 0.16.1\r\nnose: 1.3.3\r\nCython: 0.22\r\nnumpy: 1.9.2\r\nscipy: None\r\nstatsmodels: None\r\nIPython: 2.1.0\r\nsphinx: None\r\npatsy: None\r\ndateutil: 2.4.2\r\npytz: 2015.4\r\nbottleneck: None\r\ntables: 3.2.0\r\nnumexpr: 2.4\r\nmatplotlib: None\r\nopenpyxl: None\r\nxlrd: 0.9.3\r\nxlwt: None\r\nxlsxwriter: None\r\nlxml: None\r\nbs4: 4.3.2\r\nhtml5lib: None\r\nhttplib2: 0.9\r\napiclient: None\r\nsqlalchemy: 0.9.8\r\npymysql: None\r\npsycopg2: 2.6 (dt dec pq3 ext lo64)\r\n\r\nHDF5 Versions:\r\n~/user$ dpkg -l | grep \"hdf5\"\r\nii  hdf5-helpers                   1.8.13+docs-15              amd64        Hierarchical Data Format 5 (HDF5) - Helper tools\r\nii  hdf5-tools                     1.8.13+docs-15              amd64        Hierarchical Data Format 5 (HDF5) - Runtime tools\r\nrc  libhdf5-7:amd64                1.8.12+docs-1               amd64        Hierarchical Data Format 5 (HDF5) - runtime files - serial version\r\nii  libhdf5-8:amd64                1.8.13+docs-15              amd64        Hierarchical Data Format 5 (HDF5) - runtime files - serial version\r\nrc  libhdf5-cpp-7:amd64            1.8.12+docs-1               amd64        Hierarchical Data Format 5 (HDF5) - C++ libraries\r\nii  libhdf5-cpp-8:amd64            1.8.13+docs-15              amd64        Hierarchical Data Format 5 (HDF5) - C++ libraries\r\nii  libhdf5-dev                    1.8.13+docs-15              amd64        Hierarchical Data Format 5 (HDF5) - development files - serial version\r\n```\r\nA script to reproduce:\r\n```\r\nimport os\r\nimport pandas as pd\r\n\r\ndir = 'test_files'\r\nfile = './hdf5_test.h5'\r\n\r\n\r\ndef write_to_file(data):\r\n    store = pd.HDFStore(file, 'a')\r\n    for tablename, d in data.items():\r\n        frame = pd.DataFrame(series)\r\n        print(\"Appending data from {0} to {1}\".format(\r\n            series.index[0], series.index[-1]))\r\n        store.append(tablename, frame)\r\n    store.close()\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    if os.path.isfile(file):\r\n        os.remove(file)\r\n    files = os.listdir(dir)\r\n    for f in files:\r\n        series = pd.read_pickle(os.path.join(dir, f))\r\n        data = {'testtable': series}\r\n        print(\"Writing data: %s\" % f)\r\n        write_to_file(data)\r\n```\r\n\r\nFiles are zipped here (pandas pickles labelled 00-02):\r\nhttp://s000.tinyupload.com/?file_id=60238823358379433453\r\n\r\nHere is a pastebin of the data *where segfault is occuring* from the example in csv format:\r\nhttp://pastebin.com/FRsygCUG\r\n*note, you may need the actual files to reproduce this, but as you can see from the pastebin there is nothing weird going on with the data.*\r\n\r\nIt is trying to fill the the following gap in the original data:\r\n`2011-01-04 17:55:00` to` 2011-01-05 22:15:00`\r\nwith an append which results in a segfault\r\n\r\nScript output:\r\n```\r\nWriting data: 00\r\nAppending data from 2011-01-03 13:40:00 to 2011-01-04 17:55:00\r\nWriting data: 01\r\nAppending data from 2011-01-05 22:15:00 to 2011-01-07 02:30:00\r\nWriting data: 02\r\nAppending data from 2011-01-04 18:00:00 to 2011-01-05 22:15:00\r\nSegmentation fault\r\n```"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/104050984","html_url":"https://github.com/pydata/pandas/issues/10180#issuecomment-104050984","issue_url":"https://api.github.com/repos/pydata/pandas/issues/10180","id":104050984,"user":{"login":"bastianl","id":8620425,"avatar_url":"https://avatars.githubusercontent.com/u/8620425?v=3","gravatar_id":"","url":"https://api.github.com/users/bastianl","html_url":"https://github.com/bastianl","followers_url":"https://api.github.com/users/bastianl/followers","following_url":"https://api.github.com/users/bastianl/following{/other_user}","gists_url":"https://api.github.com/users/bastianl/gists{/gist_id}","starred_url":"https://api.github.com/users/bastianl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bastianl/subscriptions","organizations_url":"https://api.github.com/users/bastianl/orgs","repos_url":"https://api.github.com/users/bastianl/repos","events_url":"https://api.github.com/users/bastianl/events{/privacy}","received_events_url":"https://api.github.com/users/bastianl/received_events","type":"User","site_admin":false},"created_at":"2015-05-20T21:58:28Z","updated_at":"2015-05-20T21:58:28Z","body":"I'm pretty sure the order in which the appends occur in this case matter. If you just write the pastebin as `csv` in one go you won't see the error. I saw something similar when coerced the 3 files into a dataframe (or using store.overwrite) in memory, before appending. (also, I think the HDFStore needs to be opened in 'a' mode, or it will simply overwrite your data every time).\r\n```\r\nWriting data: 00\r\n<class 'pandas.core.frame.DataFrame'>\r\nDatetimeIndex: 340 entries, 2011-01-03 13:40:00 to 2011-01-04 17:55:00\r\nFreq: 5T\r\nData columns (total 1 columns):\r\nnormal    340 non-null float64\r\ndtypes: float64(1)Appending data from 2011-01-03 13:40:00 to 2011-01-04 17:55:00\r\nWriting data: 01\r\n<class 'pandas.core.frame.DataFrame'>\r\nDatetimeIndex: 340 entries, 2011-01-05 22:15:00 to 2011-01-07 02:30:00\r\nFreq: 5T\r\nData columns (total 1 columns):\r\nnormal    340 non-null float64\r\ndtypes: float64(1)Appending data from 2011-01-05 22:15:00 to 2011-01-07 02:30:00\r\nWriting data: 02\r\n<class 'pandas.core.frame.DataFrame'>\r\nDatetimeIndex: 340 entries, 2011-01-04 18:00:00 to 2011-01-05 22:15:00\r\nFreq: 5T\r\nData columns (total 1 columns):\r\nnormal    340 non-null float64\r\ndtypes: float64(1)Appending data from 2011-01-04 18:00:00 to 2011-01-05 22:15:00\r\nSegmentation fault\r\n```\r\n@jreback going to try on mac os x when I get a chance"}}