{"action":"created","issue":{"url":"https://api.github.com/repos/bugsnag/bugsnag-agent/issues/5","labels_url":"https://api.github.com/repos/bugsnag/bugsnag-agent/issues/5/labels{/name}","comments_url":"https://api.github.com/repos/bugsnag/bugsnag-agent/issues/5/comments","events_url":"https://api.github.com/repos/bugsnag/bugsnag-agent/issues/5/events","html_url":"https://github.com/bugsnag/bugsnag-agent/issues/5","id":92007488,"number":5,"title":"Failed requests get retried forever","user":{"login":"jesseschalken","id":1559204,"avatar_url":"https://avatars.githubusercontent.com/u/1559204?v=3","gravatar_id":"","url":"https://api.github.com/users/jesseschalken","html_url":"https://github.com/jesseschalken","followers_url":"https://api.github.com/users/jesseschalken/followers","following_url":"https://api.github.com/users/jesseschalken/following{/other_user}","gists_url":"https://api.github.com/users/jesseschalken/gists{/gist_id}","starred_url":"https://api.github.com/users/jesseschalken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jesseschalken/subscriptions","organizations_url":"https://api.github.com/users/jesseschalken/orgs","repos_url":"https://api.github.com/users/jesseschalken/repos","events_url":"https://api.github.com/users/jesseschalken/events{/privacy}","received_events_url":"https://api.github.com/users/jesseschalken/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-06-30T06:55:52Z","updated_at":"2015-07-08T00:58:14Z","closed_at":null,"body":"We were recently investigating a very high traffic bill from our datacenter to Bugsnag. It turns out that all of our servers have the following in their `/var/log/upstart/bugsnag.log`:\r\n\r\n```\r\n...\r\nCannot send request. Retrying in 5 seconds\r\nTraceback (most recent call last):\r\n  File \"/usr/bin/bugsnag-agent\", line 138, in _client\r\n    urllib2.urlopen(req).read()\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 126, in urlopen\r\n    return _opener.open(url, data, timeout)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 406, in open\r\n    response = meth(req, response)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 519, in http_response\r\n    'http', request, response, code, msg, hdrs)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 444, in error\r\n    return self._call_chain(*args)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 378, in _call_chain\r\n    result = func(*args)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 527, in http_error_default\r\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\r\nHTTPError: HTTP Error 400: Bad Request\r\ncontinuing...\r\nEnqueued 590886 bytes (119/1000)\r\nSending 612099 bytes (118/1000)\r\nCannot send request. Retrying in 5 seconds\r\nTraceback (most recent call last):\r\n  File \"/usr/bin/bugsnag-agent\", line 138, in _client\r\n    urllib2.urlopen(req).read()\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 126, in urlopen\r\n    return _opener.open(url, data, timeout)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 406, in open\r\n    response = meth(req, response)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 519, in http_response\r\n    'http', request, response, code, msg, hdrs)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 444, in error\r\n    return self._call_chain(*args)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 378, in _call_chain\r\n    result = func(*args)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 527, in http_error_default\r\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\r\nHTTPError: HTTP Error 400: Bad Request\r\ncontinuing...\r\nEnqueued 573080 bytes (119/1000)\r\n...\r\n```\r\n\r\nIt appears to have the same set of around 120 errors stuck in it's queue, and every one of the 10 threads sends one and gets the same \"400 Bad Request\" response, and requeues it and waits 5 seconds. That's 2 requests per second, or 172,800 per day, or at around 1MB per request, 172GB per server per day. With around 10 servers and running `bugsnag-agent` for around 10 days, the ~10TB bandwidth bill certainly caught us off guard.\r\n\r\n1. Is there any way to see what problem `notify.bugsnag.com` has with these requests? (they come from `bugsnag-php`)\r\n2. Can `bugsnag-agent` be made not to retry failed requests, or limit the number of retries made for each payload?"},"comment":{"url":"https://api.github.com/repos/bugsnag/bugsnag-agent/issues/comments/119388290","html_url":"https://github.com/bugsnag/bugsnag-agent/issues/5#issuecomment-119388290","issue_url":"https://api.github.com/repos/bugsnag/bugsnag-agent/issues/5","id":119388290,"user":{"login":"jesseschalken","id":1559204,"avatar_url":"https://avatars.githubusercontent.com/u/1559204?v=3","gravatar_id":"","url":"https://api.github.com/users/jesseschalken","html_url":"https://github.com/jesseschalken","followers_url":"https://api.github.com/users/jesseschalken/followers","following_url":"https://api.github.com/users/jesseschalken/following{/other_user}","gists_url":"https://api.github.com/users/jesseschalken/gists{/gist_id}","starred_url":"https://api.github.com/users/jesseschalken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jesseschalken/subscriptions","organizations_url":"https://api.github.com/users/jesseschalken/orgs","repos_url":"https://api.github.com/users/jesseschalken/repos","events_url":"https://api.github.com/users/jesseschalken/events{/privacy}","received_events_url":"https://api.github.com/users/jesseschalken/received_events","type":"User","site_admin":false},"created_at":"2015-07-08T00:58:14Z","updated_at":"2015-07-08T00:58:14Z","body":"I've been able to reproduce a 4MB payload. `Bugsnag_Client` has been configured to use batch sending (`$client->setBatchSending(true)`), which will queue up all errors and send them in a single request. If a lot of errors happen in a single PHP process, then the 524KB limit is easily exceeded.\r\n\r\nMaybe `bugsnag-php` needs to be changed to bundle multiple queued errors in batches of less than 5KB?"}}