{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/ThreatExchange/issues/79","labels_url":"https://api.github.com/repos/facebook/ThreatExchange/issues/79/labels{/name}","comments_url":"https://api.github.com/repos/facebook/ThreatExchange/issues/79/comments","events_url":"https://api.github.com/repos/facebook/ThreatExchange/issues/79/events","html_url":"https://github.com/facebook/ThreatExchange/pull/79","id":108352039,"number":79,"title":"Add support for retries","user":{"login":"mgoffin","id":260873,"avatar_url":"https://avatars.githubusercontent.com/u/260873?v=3","gravatar_id":"","url":"https://api.github.com/users/mgoffin","html_url":"https://github.com/mgoffin","followers_url":"https://api.github.com/users/mgoffin/followers","following_url":"https://api.github.com/users/mgoffin/following{/other_user}","gists_url":"https://api.github.com/users/mgoffin/gists{/gist_id}","starred_url":"https://api.github.com/users/mgoffin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgoffin/subscriptions","organizations_url":"https://api.github.com/users/mgoffin/orgs","repos_url":"https://api.github.com/users/mgoffin/repos","events_url":"https://api.github.com/users/mgoffin/events{/privacy}","received_events_url":"https://api.github.com/users/mgoffin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/facebook/ThreatExchange/labels/CLA%20Signed","name":"CLA Signed","color":"009900"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-09-25T15:50:57Z","updated_at":"2015-10-01T15:50:07Z","closed_at":"2015-10-01T15:46:14Z","pull_request":{"url":"https://api.github.com/repos/facebook/ThreatExchange/pulls/79","html_url":"https://github.com/facebook/ThreatExchange/pull/79","diff_url":"https://github.com/facebook/ThreatExchange/pull/79.diff","patch_url":"https://github.com/facebook/ThreatExchange/pull/79.patch"},"body":"Retries in `requests` is a bit of a pain, but this should expose the ability for people to retry GET, POST, and DELETE requests as often as they feel they need to in order for the request to succeed. By default we still try and fail after one attempt. This propagates into the generator so for example if we perform a query which will end up with 15 pages and a retry of `3`, we will try to fetch each page as we need to and individually they can be retried up to 3 times before failing.\r\n\r\nFailing here is defined by what requests fails on. We create a urllib3 Retry object and specify 500 and 503 responses as ones we will retry.\r\n\r\nThis PR also adds a logging feature (necessary if people are going to be doing retries and random stuff and want to inspect what pytx is doing/thinking). You can now do something like this:\r\n\r\n```python\r\nfrom pytx import setup_logger\r\n\r\nsetup_logger(\"/path/to/my/log/file\")\r\n```\r\nAfter this, any pytx code will log to this file. Any pytx errors thrown will log there, and the generator now has logging to show you the current cursors, number of results in the `data` field, and will let you know if `next` is not in the pager (meaning the generator is over) or `paging` is not in the response (meaning something entirely different?).\r\n\r\nI do not know if this will fix the issues people have mentioned about timeouts and putting things in while loops with try/except. If people can reliably generate those issues and can test with this PR to see if it fixes or helps, that would be great! If it does not (in the case where data gets submitted fine and the server returns a valid response with an internal timeout error) then we might have some other work to do but this PR could still be valid."},"comment":{"url":"https://api.github.com/repos/facebook/ThreatExchange/issues/comments/144768670","html_url":"https://github.com/facebook/ThreatExchange/pull/79#issuecomment-144768670","issue_url":"https://api.github.com/repos/facebook/ThreatExchange/issues/79","id":144768670,"user":{"login":"mgoffin","id":260873,"avatar_url":"https://avatars.githubusercontent.com/u/260873?v=3","gravatar_id":"","url":"https://api.github.com/users/mgoffin","html_url":"https://github.com/mgoffin","followers_url":"https://api.github.com/users/mgoffin/followers","following_url":"https://api.github.com/users/mgoffin/following{/other_user}","gists_url":"https://api.github.com/users/mgoffin/gists{/gist_id}","starred_url":"https://api.github.com/users/mgoffin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgoffin/subscriptions","organizations_url":"https://api.github.com/users/mgoffin/orgs","repos_url":"https://api.github.com/users/mgoffin/repos","events_url":"https://api.github.com/users/mgoffin/events{/privacy}","received_events_url":"https://api.github.com/users/mgoffin/received_events","type":"User","site_admin":false},"created_at":"2015-10-01T15:50:07Z","updated_at":"2015-10-01T15:50:07Z","body":"If you do that, log_file will be None, and no logging will be setup."}}