{"action":"created","issue":{"url":"https://api.github.com/repos/tbarbugli/cassandra_snapshotter/issues/54","labels_url":"https://api.github.com/repos/tbarbugli/cassandra_snapshotter/issues/54/labels{/name}","comments_url":"https://api.github.com/repos/tbarbugli/cassandra_snapshotter/issues/54/comments","events_url":"https://api.github.com/repos/tbarbugli/cassandra_snapshotter/issues/54/events","html_url":"https://github.com/tbarbugli/cassandra_snapshotter/issues/54","id":108343861,"number":54,"title":"Strange boto error on `upload_node_backups`","user":{"login":"winks","id":44402,"avatar_url":"https://avatars.githubusercontent.com/u/44402?v=3","gravatar_id":"","url":"https://api.github.com/users/winks","html_url":"https://github.com/winks","followers_url":"https://api.github.com/users/winks/followers","following_url":"https://api.github.com/users/winks/following{/other_user}","gists_url":"https://api.github.com/users/winks/gists{/gist_id}","starred_url":"https://api.github.com/users/winks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/winks/subscriptions","organizations_url":"https://api.github.com/users/winks/orgs","repos_url":"https://api.github.com/users/winks/repos","events_url":"https://api.github.com/users/winks/events{/privacy}","received_events_url":"https://api.github.com/users/winks/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-25T15:06:43Z","updated_at":"2015-09-25T19:21:53Z","closed_at":null,"body":"Hi,\r\nI just started using cassandra_snapshotter and I got a similar error twice when running it on my cluster (4 nodes, each seems to have ~18GB of data)\r\n\r\n```\r\n[172.31.10.42] out: boto.exception.S3ResponseError: S3ResponseError: 200 OK\r\n````\r\n\r\n[Full log of the run is here](https://gist.github.com/winks/989ff567c11879646d3e)\r\n\r\nHere are outputs of `s3cmd du` (converted to GB) on the `path/CLUSTERNAME/IP` folders in s3, as you can see, the 1st pair is missing some data while on my first try (2nd pair) it's not so obvious.\r\n\r\n``` \r\n17.74581958632916212081 run 1, node 1\r\n12.03844990301877260208 run2\r\n\r\n17.30710690561681985855 run 1, node 2\r\n17.32890863530337810516 run 2\r\n\r\n18.47752564307302236557 run 1, node 3\r\n18.50679215136915445327 run 2\r\n\r\n17.72296549659222364425 run 1, node 4\r\n17.74818217754364013671 run 2\r\n```\r\n\r\n**TLDR:** I think either there's something wrong with S3 atm, or cassandra_snapshotter should really retry to upload the snapshot if there's a boto error...\r\n\r\nNot sure if [this issue](https://github.com/boto/boto/issues/840) is related."},"comment":{"url":"https://api.github.com/repos/tbarbugli/cassandra_snapshotter/issues/comments/143330131","html_url":"https://github.com/tbarbugli/cassandra_snapshotter/issues/54#issuecomment-143330131","issue_url":"https://api.github.com/repos/tbarbugli/cassandra_snapshotter/issues/54","id":143330131,"user":{"login":"tbarbugli","id":88735,"avatar_url":"https://avatars.githubusercontent.com/u/88735?v=3","gravatar_id":"","url":"https://api.github.com/users/tbarbugli","html_url":"https://github.com/tbarbugli","followers_url":"https://api.github.com/users/tbarbugli/followers","following_url":"https://api.github.com/users/tbarbugli/following{/other_user}","gists_url":"https://api.github.com/users/tbarbugli/gists{/gist_id}","starred_url":"https://api.github.com/users/tbarbugli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbarbugli/subscriptions","organizations_url":"https://api.github.com/users/tbarbugli/orgs","repos_url":"https://api.github.com/users/tbarbugli/repos","events_url":"https://api.github.com/users/tbarbugli/events{/privacy}","received_events_url":"https://api.github.com/users/tbarbugli/received_events","type":"User","site_admin":false},"created_at":"2015-09-25T19:21:53Z","updated_at":"2015-09-25T19:21:53Z","body":"I doubt S3 is having an outage, there is probably something else wrong\nhere. Do you specify the S3 region name? What region are you using?\n\nOn Fri, Sep 25, 2015 at 11:35 AM Florian Anderiasch <\nnotifications@github.com> wrote:\n\n> As I said, I did 2 runs and both failed (but on only one host each, and it\n> were different ones).\n> I wanted to wait until tomorrow to run another one, will come back with\n> results then.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/tbarbugli/cassandra_snapshotter/issues/54#issuecomment-143297012>\n> .\n>\n"}}