{"action":"created","issue":{"url":"https://api.github.com/repos/ropensci/fishbaseapi/issues/68","labels_url":"https://api.github.com/repos/ropensci/fishbaseapi/issues/68/labels{/name}","comments_url":"https://api.github.com/repos/ropensci/fishbaseapi/issues/68/comments","events_url":"https://api.github.com/repos/ropensci/fishbaseapi/issues/68/events","html_url":"https://github.com/ropensci/fishbaseapi/issues/68","id":106585432,"number":68,"title":"invalid content-length header for api requests with limit > 1000","user":{"login":"jhpoelen","id":1084872,"avatar_url":"https://avatars.githubusercontent.com/u/1084872?v=3","gravatar_id":"","url":"https://api.github.com/users/jhpoelen","html_url":"https://github.com/jhpoelen","followers_url":"https://api.github.com/users/jhpoelen/followers","following_url":"https://api.github.com/users/jhpoelen/following{/other_user}","gists_url":"https://api.github.com/users/jhpoelen/gists{/gist_id}","starred_url":"https://api.github.com/users/jhpoelen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhpoelen/subscriptions","organizations_url":"https://api.github.com/users/jhpoelen/orgs","repos_url":"https://api.github.com/users/jhpoelen/repos","events_url":"https://api.github.com/users/jhpoelen/events{/privacy}","received_events_url":"https://api.github.com/users/jhpoelen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-09-15T15:46:45Z","updated_at":"2015-09-16T18:16:40Z","closed_at":null,"body":"hey y'all:\r\n\r\nI've been poking around the fishbase api . . . thanks for building it! \r\n\r\nI am observing a strange content length issue that occurs for larger requests (I am interested in getting *all* data).\r\n\r\nTo reproduce:\r\n```wget http://fishbase.ropensci.org/fooditems?limit=1000&offset=1600```\r\n\r\nExpected result:\r\nSuccessful download of first 1000 records in fooditems table.\r\n\r\nActual result:\r\nFor some reason the content-length is set too high, 'causing wget (and any other http client that checks the content-length) to thing the connection is closed unexpectedly (see log trace below). \r\n\r\nNote that this issue does not occur when manually downloading the data using a browser, even though the content-length is off. Also, the issue does not seem to occur when smaller batches are used (e.g. limit=100). \r\n\r\nHope this helps,\r\n-jorrit\r\n\r\n````\r\nwget \"http://fishbase.ropensci.org/fooditems?limit=1000&offset=1600\"\r\n--2015-09-15 08:39:34--  http://fishbase.ropensci.org/fooditems?limit=1000&offset=1600\r\nResolving fishbase.ropensci.org... 198.199.104.22\r\nConnecting to fishbase.ropensci.org|198.199.104.22|:80... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 895255 (874K) [application/json]\r\nSaving to: `fooditems?limit=1000&offset=1600'\r\n\r\n 7% [=>                                     ] 65,793       418K/s   in 0.2s    \r\n\r\n2015-09-15 08:39:34 (418 KB/s) - Connection closed at byte 65793. Retrying.\r\n\r\n--2015-09-15 08:39:35--  (try: 2)  http://fishbase.ropensci.org/fooditems?limit=1000&offset=1600\r\nConnecting to fishbase.ropensci.org|198.199.104.22|:80... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 895255 (874K) [application/json]\r\nSaving to: `fooditems?limit=1000&offset=1600'\r\n\r\n 7% [=>                                     ] 65,793       401K/s   in 0.2s   \r\n````\r\n "},"comment":{"url":"https://api.github.com/repos/ropensci/fishbaseapi/issues/comments/140828609","html_url":"https://github.com/ropensci/fishbaseapi/issues/68#issuecomment-140828609","issue_url":"https://api.github.com/repos/ropensci/fishbaseapi/issues/68","id":140828609,"user":{"login":"jhpoelen","id":1084872,"avatar_url":"https://avatars.githubusercontent.com/u/1084872?v=3","gravatar_id":"","url":"https://api.github.com/users/jhpoelen","html_url":"https://github.com/jhpoelen","followers_url":"https://api.github.com/users/jhpoelen/followers","following_url":"https://api.github.com/users/jhpoelen/following{/other_user}","gists_url":"https://api.github.com/users/jhpoelen/gists{/gist_id}","starred_url":"https://api.github.com/users/jhpoelen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhpoelen/subscriptions","organizations_url":"https://api.github.com/users/jhpoelen/orgs","repos_url":"https://api.github.com/users/jhpoelen/repos","events_url":"https://api.github.com/users/jhpoelen/events{/privacy}","received_events_url":"https://api.github.com/users/jhpoelen/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T18:16:40Z","updated_at":"2015-09-16T18:16:40Z","body":"Unfortunately, it seems that the issue is still occurring. \r\n\r\nTo reproduce:\r\n\r\n```\r\nwget \"http://fishbase.ropensci.org/species?limit=5000&offset=0\"\r\n--2015-09-16 11:14:01--  http://fishbase.ropensci.org/species?limit=5000&offset=0\r\nResolving fishbase.ropensci.org... 198.199.104.22\r\nConnecting to fishbase.ropensci.org|198.199.104.22|:80... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 14377534 (14M) [application/json]\r\nSaving to: `species?limit=5000&offset=0'\r\n\r\n45% [===============================>                                       ] 6,576,816    394K/s   in 27s     \r\n\r\n2015-09-16 11:14:37 (235 KB/s) - Connection closed at byte 6576816. Retrying.\r\n\r\n--2015-09-16 11:14:38--  (try: 2)  http://fishbase.ropensci.org/species?limit=5000&offset=0\r\nConnecting to fishbase.ropensci.org|198.199.104.22|:80... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 14377534 (14M) [application/json]\r\nSaving to: `species?limit=5000&offset=0'\r\n\r\n 9% [======>                                                                ] 1,428,432    382K/s  eta 39s \r\n```"}}