{"action":"created","issue":{"url":"https://api.github.com/repos/astropy/astropy/issues/4307","labels_url":"https://api.github.com/repos/astropy/astropy/issues/4307/labels{/name}","comments_url":"https://api.github.com/repos/astropy/astropy/issues/4307/comments","events_url":"https://api.github.com/repos/astropy/astropy/issues/4307/events","html_url":"https://github.com/astropy/astropy/pull/4307","id":116445957,"number":4307,"title":"Saving large table FITS may lead to a truncated file","user":{"login":"yannick1974","id":628327,"avatar_url":"https://avatars.githubusercontent.com/u/628327?v=3","gravatar_id":"","url":"https://api.github.com/users/yannick1974","html_url":"https://github.com/yannick1974","followers_url":"https://api.github.com/users/yannick1974/followers","following_url":"https://api.github.com/users/yannick1974/following{/other_user}","gists_url":"https://api.github.com/users/yannick1974/gists{/gist_id}","starred_url":"https://api.github.com/users/yannick1974/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yannick1974/subscriptions","organizations_url":"https://api.github.com/users/yannick1974/orgs","repos_url":"https://api.github.com/users/yannick1974/repos","events_url":"https://api.github.com/users/yannick1974/events{/privacy}","received_events_url":"https://api.github.com/users/yannick1974/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/astropy/astropy/labels/Affects-release","name":"Affects-release","color":"0052cc"},{"url":"https://api.github.com/repos/astropy/astropy/labels/Bug","name":"Bug","color":"eb6420"},{"url":"https://api.github.com/repos/astropy/astropy/labels/io.fits","name":"io.fits","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/astropy/astropy/milestones/31","html_url":"https://github.com/astropy/astropy/milestones/v1.0.7","labels_url":"https://api.github.com/repos/astropy/astropy/milestones/31/labels","id":1371114,"number":31,"title":"v1.0.7","description":"","creator":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":6,"state":"open","created_at":"2015-10-22T19:20:21Z","updated_at":"2015-11-16T15:04:10Z","due_on":null,"closed_at":null},"comments":19,"created_at":"2015-11-11T23:40:45Z","updated_at":"2015-11-16T15:04:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/astropy/astropy/pulls/4307","html_url":"https://github.com/astropy/astropy/pull/4307","diff_url":"https://github.com/astropy/astropy/pull/4307.diff","patch_url":"https://github.com/astropy/astropy/pull/4307.patch"},"body":"While handling large catalogue FITS files, I sometimes encounter problems when saving them: the newly saved file is truncated whereas no error is raised at saving time. For instance, I have a 37 GiB file and if I open it and save it just after, this leads to a 2 GiB file.\r\n\r\nI tried to write a script to generate a big FITS file and expose the bug. During this, I found that if read a big file and count the rows before saving it, the newly save file is OK.\r\n\r\nHere is the script. Of course, you need a lot of memory and a lot of disk space to run it.\r\n\r\n```python\r\nimport os\r\n\r\nimport numpy as np\r\nfrom astropy.io import fits\r\n\r\n# Let's generate a BIG fits file (~25 GiB).\r\nprint(\"> Generating a 500x13,000,000 random table FITS file.\")\r\nlist_cols = []\r\nfor i in range(500):\r\n    list_cols.append(\r\n        fits.Column(\r\n            name=\"{}\".format(i),\r\n            format='E',\r\n            array=np.random.random(13000000)\r\n        )\r\n    )\r\ncols = fits.ColDefs(list_cols)\r\nhdu = fits.BinTableHDU.from_columns(cols)\r\nhdu.writeto(\"bigfile.fits\")\r\nprint(\"> bigfile.fits file saved.\")\r\n\r\n# If we count read the file and count the rows before saving it to another\r\n# file, everything is OK.\r\nprint(\"> Read. Count rows. Save.\")\r\nhdu_list = fits.open(\"bigfile.fits\")\r\nprint(\"bigfile.fits: {} rows\".format(len(hdu_list[1].data)))\r\nhdu_list.writeto(\"bigfile2.fits\")\r\nprint(\"> bigfile2.fits saved.\")\r\n\r\nprint(\"> Comparing the file sizes.\")\r\nprint(\"bigfile.fits: {}\".format(os.stat(\"bigfile.fits\").st_size))\r\nprint(\"bigfile2.fits: {}\".format(os.stat(\"bigfile2.fits\").st_size))\r\n\r\nprint(\"Reading bigfile2.fits and counting the rows.\")\r\nhdu_list = fits.open(\"bigfile2.fits\")\r\nprint(\"bigfile2.fits: {} rows\".format(len(hdu_list[1].data)))\r\n\r\n# If we read the file and right after save it to another file, the later is\r\n# truncated.\r\nprint(\"> Read the file and save it to another file just after.\")\r\nhdu_list = fits.open(\"bigfile.fits\")\r\nhdu_list.writeto(\"bigfile3.fits\")\r\nprint(\"> bigfile3.fits saved.\")\r\n\r\nprint(\"> Comparing the file sizes.\")\r\nprint(\"bigfile.fits: {}\".format(os.stat(\"bigfile.fits\").st_size))\r\nprint(\"bigfile3.fits: {}\".format(os.stat(\"bigfile3.fits\").st_size))\r\n\r\nprint(\"> Reading bigfile3.fits produce a warning.\")\r\nhdu_list = fits.open(\"bigfile3.fits\")  # Produces a warning.\r\nprint(\"> Accessing the data raises an error.\")\r\nprint(\"bigfile3.fits: {} rows\".format(len(hdu_list[1].data)))  # Fails.\r\n```\r\n\r\nThe output of the script is:\r\n```\r\n> Generating a 500x13,000,000 random table FITS file.\r\n> bigfile.fits file saved.\r\n> Read. Count rows. Save.\r\nbigfile.fits: 13000000 rows\r\n> bigfile2.fits saved.\r\n> Comparing the file sizes.\r\nbigfile.fits: 26000087040\r\nbigfile2.fits: 26000087040\r\nReading bigfile2.fits and counting the rows.\r\nbigfile2.fits: 13000000 rows\r\n> Read the file and save it to another file just after.\r\n> bigfile3.fits saved.\r\n> Comparing the file sizes.\r\nbigfile.fits: 26000087040\r\nbigfile3.fits: 2147565952\r\n> Reading bigfile3.fits produce a warning.\r\nWARNING: File may have been truncated: actual file length (2147565952) is smaller than the expected size (26000087040) [astropy.io.fits.file]\r\n> Accessing the data raises an error.\r\nTraceback (most recent call last):\r\n  File \"/opt/anaconda3/lib/python3.4/site-packages/astropy/utils/decorators.py\", line 339, in __get__\r\n    return obj.__dict__[self._key]\r\nKeyError: 'data'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"big_table_FITS_bug.py\", line 52, in <module>\r\n    print(\"bigfile3.fits: {} rows\".format(len(hdu_list[1].data)))  # Fails.\r\n  File \"/opt/anaconda3/lib/python3.4/site-packages/astropy/utils/decorators.py\", line 341, in __get__\r\n    val = self._fget(obj)\r\n  File \"/opt/anaconda3/lib/python3.4/site-packages/astropy/io/fits/hdu/table.py\", line 399, in data\r\n    data = self._get_tbdata()\r\n  File \"/opt/anaconda3/lib/python3.4/site-packages/astropy/io/fits/hdu/table.py\", line 168, in _get_tbdata\r\n    self._data_offset)\r\n  File \"/opt/anaconda3/lib/python3.4/site-packages/astropy/io/fits/hdu/base.py\", line 556, in _get_raw_data\r\n    return self._file.readarray(offset=offset, dtype=code, shape=shape)\r\n  File \"/opt/anaconda3/lib/python3.4/site-packages/astropy/io/fits/file.py\", line 271, in readarray\r\n    buffer=self._mmap)\r\nTypeError: buffer is too small for requested array\r\n```\r\n\r\nThe main problem is that astropy does not raise an error when the saving leads to a truncated file. The second problem is that astropy should be able to handle large files.\r\n\r\nI have no idea on how to debug this but I'm willing to help.\r\n\r\nYannick"},"comment":{"url":"https://api.github.com/repos/astropy/astropy/issues/comments/157060630","html_url":"https://github.com/astropy/astropy/pull/4307#issuecomment-157060630","issue_url":"https://api.github.com/repos/astropy/astropy/issues/4307","id":157060630,"user":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"created_at":"2015-11-16T15:04:30Z","updated_at":"2015-11-16T15:04:30Z","body":"@juliantaylor @yannick1974 Excellent, thanks for confirming."}}