{"action":"created","issue":{"url":"https://api.github.com/repos/jparyani/pycapnp/issues/82","labels_url":"https://api.github.com/repos/jparyani/pycapnp/issues/82/labels{/name}","comments_url":"https://api.github.com/repos/jparyani/pycapnp/issues/82/comments","events_url":"https://api.github.com/repos/jparyani/pycapnp/issues/82/events","html_url":"https://github.com/jparyani/pycapnp/issues/82","id":103930592,"number":82,"title":"KjException: failed: miniposix::read(fd, pos, max - pos): Bad file descriptor;","user":{"login":"ejhumphrey","id":926379,"avatar_url":"https://avatars.githubusercontent.com/u/926379?v=3","gravatar_id":"","url":"https://api.github.com/users/ejhumphrey","html_url":"https://github.com/ejhumphrey","followers_url":"https://api.github.com/users/ejhumphrey/followers","following_url":"https://api.github.com/users/ejhumphrey/following{/other_user}","gists_url":"https://api.github.com/users/ejhumphrey/gists{/gist_id}","starred_url":"https://api.github.com/users/ejhumphrey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejhumphrey/subscriptions","organizations_url":"https://api.github.com/users/ejhumphrey/orgs","repos_url":"https://api.github.com/users/ejhumphrey/repos","events_url":"https://api.github.com/users/ejhumphrey/events{/privacy}","received_events_url":"https://api.github.com/users/ejhumphrey/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-30T15:27:07Z","updated_at":"2015-11-28T02:09:10Z","closed_at":null,"body":"Attempts to read large integer lists (thousands of elements) are causing a KjException via the pycapnp library, but parse fine in C++. Not sure what's going wrong, and hoping the example repository I threw together can help troubleshoot the issue. Entirely possible I'm doing something incorrectly, but this seemed like the place to start since the exception is getting thrown in Python:\r\n\r\nhttps://github.com/ejhumphrey/capn-demo\r\n\r\nAny help would be greatly appreciated!\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"test_pycapnp.py\", line 26, in <module>\r\n    test_reading(args.num_elements)\r\n  File \"test_pycapnp.py\", line 16, in test_reading\r\n    np.asarray(x.values, dtype=DTYPE),\r\n  File \"capnp/lib/capnp.pyx\", line 1022, in capnp.lib.capnp._DynamicStructReader.__getattr__ (capnp/lib/capnp.cpp:22190)\r\ncapnp.lib.capnp.KjException: src/kj/io.c++:259: failed: miniposix::read(fd, pos, max - pos): Bad file descriptor; fd = 3\r\nstack: 0x1060d1850 0x1060d5379 0x1060d7a0b 0x1060d70ea 0x1060d67cf 0x1060d65f5 0x10602129c 0x1060d64b1 0x106020329 0x10600b751 0x1060133be 0x10603917a 0x10603f123 0x105d100b5 0x105d98f26 0x1059a37c4\r\nlibc++abi.dylib: terminating with uncaught exception of type kj::ExceptionImpl: src/kj/io.c++:259: failed: miniposix::read(fd, pos, max - pos): Bad file descriptor; fd = 3\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/jparyani/pycapnp/issues/comments/160238976","html_url":"https://github.com/jparyani/pycapnp/issues/82#issuecomment-160238976","issue_url":"https://api.github.com/repos/jparyani/pycapnp/issues/82","id":160238976,"user":{"login":"drakedevel","id":403067,"avatar_url":"https://avatars.githubusercontent.com/u/403067?v=3","gravatar_id":"","url":"https://api.github.com/users/drakedevel","html_url":"https://github.com/drakedevel","followers_url":"https://api.github.com/users/drakedevel/followers","following_url":"https://api.github.com/users/drakedevel/following{/other_user}","gists_url":"https://api.github.com/users/drakedevel/gists{/gist_id}","starred_url":"https://api.github.com/users/drakedevel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drakedevel/subscriptions","organizations_url":"https://api.github.com/users/drakedevel/orgs","repos_url":"https://api.github.com/users/drakedevel/repos","events_url":"https://api.github.com/users/drakedevel/events{/privacy}","received_events_url":"https://api.github.com/users/drakedevel/received_events","type":"User","site_admin":false},"created_at":"2015-11-28T02:09:10Z","updated_at":"2015-11-28T02:09:10Z","body":"I just ran into this bug! The issue is that `read` and `read_packed` don't actually read the entire contents of the file eagerly -- they will read more data in as needed. The real bug is that pycapnp doesn't hold a reference to the file object you pass in, so it gets freed after the `read` call (closing the fd) and the underlying C++ object is left with a reference to a now-invalid file descriptor. As soon as a field is accessed past the end of the initial buffered read it will crash (or worse, read from a random subsequently-opened file). This is the relevant strace from my system:\r\n\r\n```\r\nopen(\"data/tmp.cpn\", O_RDONLY|O_CLOEXEC) = 7\r\nfstat(7, {st_mode=S_IFREG|0644, st_size=24726, ...}) = 0\r\nioctl(7, TCGETS, 0x7ffcfcfbaf30)        = -1 ENOTTY (Inappropriate ioctl for device)\r\nfstat(7, {st_mode=S_IFREG|0644, st_size=24726, ...}) = 0\r\nlseek(7, 0, SEEK_CUR)                   = 0\r\nioctl(7, TCGETS, 0x7ffcfcfbae90)        = -1 ENOTTY (Inappropriate ioctl for device)\r\nlseek(7, 0, SEEK_CUR)                   = 0\r\nread(7, \"\\21\\1\\4\\3\\211\\23@\\1\\21\\1\\17A\\4\\1\\21\\2\\1q\\1\\2048\\1\\20\\1\\21\\2\\3\\21\\4\\5\\21\\6\"..., 8192) = 8192\r\nclose(7)                                = 0\r\nread(7, 0x55d60d17eee0, 8192)           = -1 EBADF (Bad file descriptor)\r\n```\r\n\r\nThe easiest workaround is to hold the file open while using the capnp objects. I modified your example, wrapping the body of `test_read` with `with open(FNAME, 'r') as f:` and the tests pass with arbitrarily large numbers."}}