{"action":"created","issue":{"url":"https://api.github.com/repos/spacetelescope/pyasdf/issues/144","labels_url":"https://api.github.com/repos/spacetelescope/pyasdf/issues/144/labels{/name}","comments_url":"https://api.github.com/repos/spacetelescope/pyasdf/issues/144/comments","events_url":"https://api.github.com/repos/spacetelescope/pyasdf/issues/144/events","html_url":"https://github.com/spacetelescope/pyasdf/pull/144","id":94150765,"number":144,"title":"Add index","user":{"login":"mdboom","id":38294,"avatar_url":"https://avatars.githubusercontent.com/u/38294?v=3","gravatar_id":"","url":"https://api.github.com/users/mdboom","html_url":"https://github.com/mdboom","followers_url":"https://api.github.com/users/mdboom/followers","following_url":"https://api.github.com/users/mdboom/following{/other_user}","gists_url":"https://api.github.com/users/mdboom/gists{/gist_id}","starred_url":"https://api.github.com/users/mdboom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdboom/subscriptions","organizations_url":"https://api.github.com/users/mdboom/orgs","repos_url":"https://api.github.com/users/mdboom/repos","events_url":"https://api.github.com/users/mdboom/events{/privacy}","received_events_url":"https://api.github.com/users/mdboom/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-07-09T21:00:59Z","updated_at":"2015-07-17T20:45:38Z","closed_at":"2015-07-17T17:00:28Z","pull_request":{"url":"https://api.github.com/repos/spacetelescope/pyasdf/pulls/144","html_url":"https://github.com/spacetelescope/pyasdf/pull/144","diff_url":"https://github.com/spacetelescope/pyasdf/pull/144.diff","patch_url":"https://github.com/spacetelescope/pyasdf/pull/144.patch"},"body":"This adds a block index to the end of the file.  ~~I'll have to document this over in the spec, but it's basically:\r\n\r\n```\r\n�IDX # ASDF BLOCK INDEX\r\n8192\r\n16384\r\n24576\r\n25142\r\n```\r\n\r\nwhere the numbers are the byte index of each block.~~\r\n\r\nTo the extent that this index format is \"versioned\" and may change in the future, I think it should follow the version of the file format as a whole (i.e. unlike tag schemas, I don't think this should be individually versioned).\r\n\r\nThe way the index is read is a little bit heuristic and magical (basically reading chunk-by-chunk backward until finding the index header followed by a list of numbers -- anything else is just rejected and we fall back to the \"skipping through\" method of finding blocks).  There's still the \"it's technically possible, but highly unlikely, that someone's data would look like an index\" problem.  I don't know how much I should worry about that.  I thought about using binary values for the indices, but I felt like they would have a greater likelihood of looking like real data, and I don't think there's a real performance advantage to storing these in binary.\r\n\r\nCc: @embray"},"comment":{"url":"https://api.github.com/repos/spacetelescope/pyasdf/issues/comments/122408790","html_url":"https://github.com/spacetelescope/pyasdf/pull/144#issuecomment-122408790","issue_url":"https://api.github.com/repos/spacetelescope/pyasdf/issues/144","id":122408790,"user":{"login":"mdboom","id":38294,"avatar_url":"https://avatars.githubusercontent.com/u/38294?v=3","gravatar_id":"","url":"https://api.github.com/users/mdboom","html_url":"https://github.com/mdboom","followers_url":"https://api.github.com/users/mdboom/followers","following_url":"https://api.github.com/users/mdboom/following{/other_user}","gists_url":"https://api.github.com/users/mdboom/gists{/gist_id}","starred_url":"https://api.github.com/users/mdboom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdboom/subscriptions","organizations_url":"https://api.github.com/users/mdboom/orgs","repos_url":"https://api.github.com/users/mdboom/repos","events_url":"https://api.github.com/users/mdboom/events{/privacy}","received_events_url":"https://api.github.com/users/mdboom/received_events","type":"User","site_admin":false},"created_at":"2015-07-17T20:45:38Z","updated_at":"2015-07-17T20:45:38Z","body":"@embray: Very good question, and I should have thought of it before I hastily merged.  The benefits are actually fairly small in my measurements.\r\n\r\nI put a timing script here:\r\n\r\nhttps://gist.github.com/mdboom/5769234ab3f96054adc6\r\n\r\n(Note, you'll need #149 to run the benchmark).\r\n\r\nIn the following results the first number is the number of blocks in the file.  The timing is how long it takes to read the last block.  On my Linux laptop with an SSD, I get:\r\n\r\n```\r\n10: Indexed: 0.00803089141846, Unindexed: 0.0077440738678\r\n100: Indexed: 0.0669429302216, Unindexed: 0.0636179447174\r\n1000: Indexed: 0.649363040924, Unindexed: 0.639756917953\r\n10000: Indexed: 6.90504193306, Unindexed: 8.13161206245\r\n```\r\n\r\nSo, until you get to somewhere between 1,000 and 10,000 blocks, there is actually a slight penalty to using the index.\r\n\r\nOf course, this all may behave quite differently on a spinning rust drive (which I don't have with me but can test next week) or over a network.\r\n\r\nThere's probably a safe heuristic here to not output an index when the number of blocks is quite small."}}