{"action":"created","issue":{"url":"https://api.github.com/repos/buffer/thug/issues/127","labels_url":"https://api.github.com/repos/buffer/thug/issues/127/labels{/name}","comments_url":"https://api.github.com/repos/buffer/thug/issues/127/comments","events_url":"https://api.github.com/repos/buffer/thug/issues/127/events","html_url":"https://github.com/buffer/thug/issues/127","id":99426167,"number":127,"title":"Recursive redirect processing until analysis timeout or threshold reached","user":{"login":"wojtossfm","id":4896604,"avatar_url":"https://avatars.githubusercontent.com/u/4896604?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtossfm","html_url":"https://github.com/wojtossfm","followers_url":"https://api.github.com/users/wojtossfm/followers","following_url":"https://api.github.com/users/wojtossfm/following{/other_user}","gists_url":"https://api.github.com/users/wojtossfm/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtossfm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtossfm/subscriptions","organizations_url":"https://api.github.com/users/wojtossfm/orgs","repos_url":"https://api.github.com/users/wojtossfm/repos","events_url":"https://api.github.com/users/wojtossfm/events{/privacy}","received_events_url":"https://api.github.com/users/wojtossfm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/buffer/thug/labels/task","name":"task","color":"009800"}],"state":"open","locked":false,"assignee":{"login":"buffer","id":315927,"avatar_url":"https://avatars.githubusercontent.com/u/315927?v=3","gravatar_id":"","url":"https://api.github.com/users/buffer","html_url":"https://github.com/buffer","followers_url":"https://api.github.com/users/buffer/followers","following_url":"https://api.github.com/users/buffer/following{/other_user}","gists_url":"https://api.github.com/users/buffer/gists{/gist_id}","starred_url":"https://api.github.com/users/buffer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buffer/subscriptions","organizations_url":"https://api.github.com/users/buffer/orgs","repos_url":"https://api.github.com/users/buffer/repos","events_url":"https://api.github.com/users/buffer/events{/privacy}","received_events_url":"https://api.github.com/users/buffer/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-08-06T12:42:23Z","updated_at":"2015-08-07T07:47:01Z","closed_at":null,"body":"The following sample page causes Thug to recursively continue processing till the timeout as far as I can tell:\r\n\r\n```\r\n<html>\r\n<body>\r\n<script>\r\ntry{\r\n        //Refresh\r\n        function _refreshPage(){\r\n                var i=new Image();\r\n                function do_load(){window.location=window.location}\r\n                function do_error(){setTimeout(_refreshPage,60000)}\r\n                i.onload=do_load;\r\n                i.onerror=do_error;\r\n                i.src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAAAAAA6fptVAAAACklEQVR4nGP6DwABBQECz6AuzQAAAABJRU5ErkJggg==\";\r\n        }\r\n        setTimeout(_refreshPage,5000);\r\n}catch(ex){\r\n\r\n}\r\ntry {\r\n    //Refresh\r\n    setTimeout(function() {\r\n        document.location.href = \"/test.html?refresh_ce\";\r\n    }, 300000);\r\n} catch (ex) {}\r\ntry {\r\n    //Refresh\r\n    setTimeout(function() {\r\n        document.location.href = \"/test.html?refresh_ce\";\r\n    }, 300000);\r\n} catch (ex) {}\r\n</script>\r\n</body>\r\n</html>\r\n```\r\n\r\nReproduction steps:\r\n```\r\n# Save the test.html in the current folder\r\npython -m SimpleHTTPServer &\r\ntime python /opt/thug/src/thug.py -F -M --timeout=600 --delay=100 --verbose http://localhost:8000/test.html\r\n# logs...\r\n\r\n# Watch Thug chase its own tail?\r\n# Thug finishes analyzing only when the timeout is reached\r\nreal    10m13.069s\r\nuser    7m30.972s\r\nsys    0m3.612s\r\n```\r\n\r\nI'm not sure whether to list this as a bug or a feature request. I expected Thug to maintain some notion of visited pages and to skip revisiting of paths that were already visited. Let me know if you have trouble reproducing this, but the above example should suffice. This was observed with version 0.6.2, but as far as I looked through the commits I didn't see any changes between it and 0.6.3 that would impact this."},"comment":{"url":"https://api.github.com/repos/buffer/thug/issues/comments/128626124","html_url":"https://github.com/buffer/thug/issues/127#issuecomment-128626124","issue_url":"https://api.github.com/repos/buffer/thug/issues/127","id":128626124,"user":{"login":"buffer","id":315927,"avatar_url":"https://avatars.githubusercontent.com/u/315927?v=3","gravatar_id":"","url":"https://api.github.com/users/buffer","html_url":"https://github.com/buffer","followers_url":"https://api.github.com/users/buffer/followers","following_url":"https://api.github.com/users/buffer/following{/other_user}","gists_url":"https://api.github.com/users/buffer/gists{/gist_id}","starred_url":"https://api.github.com/users/buffer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buffer/subscriptions","organizations_url":"https://api.github.com/users/buffer/orgs","repos_url":"https://api.github.com/users/buffer/repos","events_url":"https://api.github.com/users/buffer/events{/privacy}","received_events_url":"https://api.github.com/users/buffer/received_events","type":"User","site_admin":false},"created_at":"2015-08-07T07:47:01Z","updated_at":"2015-08-07T07:47:01Z","body":"First of all thanks for your really detailed report. I can reproduce the behavior you are describing. Well in theory a real browser would behave the way Thug behaves and would keep looping until some kind of browser-defined timeout is fired. Keeping track of the already visited pages and skipping them is dangerous since it exposes to evasion (i.e. loop 2 times on the same URL and the second time change the content served and Thug is evaded). The timeout option was added long time ago so that the analyst could enforce a timeout like a real browser would do. But choosing a default value > 0 is not that simple. This the reason why I opted for no timeout at all by default and let each one decide for a specific value which fits his needs.  "}}