{"action":"closed","issue":{"url":"https://api.github.com/repos/andresriancho/w3af/issues/8698","labels_url":"https://api.github.com/repos/andresriancho/w3af/issues/8698/labels{/name}","comments_url":"https://api.github.com/repos/andresriancho/w3af/issues/8698/comments","events_url":"https://api.github.com/repos/andresriancho/w3af/issues/8698/events","html_url":"https://github.com/andresriancho/w3af/issues/8698","id":59843033,"number":8698,"title":"Too many consecutive errors","user":{"login":"andresriancho","id":865200,"avatar_url":"https://avatars.githubusercontent.com/u/865200?v=3","gravatar_id":"","url":"https://api.github.com/users/andresriancho","html_url":"https://github.com/andresriancho","followers_url":"https://api.github.com/users/andresriancho/followers","following_url":"https://api.github.com/users/andresriancho/following{/other_user}","gists_url":"https://api.github.com/users/andresriancho/gists{/gist_id}","starred_url":"https://api.github.com/users/andresriancho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andresriancho/subscriptions","organizations_url":"https://api.github.com/users/andresriancho/orgs","repos_url":"https://api.github.com/users/andresriancho/repos","events_url":"https://api.github.com/users/andresriancho/events{/privacy}","received_events_url":"https://api.github.com/users/andresriancho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/andresriancho/w3af/labels/bug","name":"bug","color":"e10c02"},{"url":"https://api.github.com/repos/andresriancho/w3af/labels/priority:high","name":"priority:high","color":"e10c02"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/andresriancho/w3af/milestones/10","html_url":"https://github.com/andresriancho/w3af/milestones/1.6.50%20-%20Bug%20fixing%20after%201.6.0","labels_url":"https://api.github.com/repos/andresriancho/w3af/milestones/10/labels","id":614510,"number":10,"title":"1.6.50 - Bug fixing after 1.6.0","description":"Bug fixes after 1.6 release, this is a MUST-HAVE since people will start using 1.6 and report severity 1 bugs which need to be fixed and pushed to all users.","creator":{"login":"andresriancho","id":865200,"avatar_url":"https://avatars.githubusercontent.com/u/865200?v=3","gravatar_id":"","url":"https://api.github.com/users/andresriancho","html_url":"https://github.com/andresriancho","followers_url":"https://api.github.com/users/andresriancho/followers","following_url":"https://api.github.com/users/andresriancho/following{/other_user}","gists_url":"https://api.github.com/users/andresriancho/gists{/gist_id}","starred_url":"https://api.github.com/users/andresriancho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andresriancho/subscriptions","organizations_url":"https://api.github.com/users/andresriancho/orgs","repos_url":"https://api.github.com/users/andresriancho/repos","events_url":"https://api.github.com/users/andresriancho/events{/privacy}","received_events_url":"https://api.github.com/users/andresriancho/received_events","type":"User","site_admin":false},"open_issues":63,"closed_issues":238,"state":"open","created_at":"2014-03-30T17:44:39Z","updated_at":"2015-03-10T15:30:03Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-03-04T18:53:46Z","updated_at":"2015-03-10T15:30:03Z","closed_at":"2015-03-10T15:30:03Z","body":"## How to reproduce\r\nWas found while trying to reproduce https://github.com/andresriancho/w3af/issues/4219 : \r\n * w3af running on my workstation\r\n * DVWA running on virtualbox\r\n\r\nI'm able to reproduce this issue in each scan\r\n\r\n> **IMPORTANT** The following error was detected by w3af and couldn't be resolved:\r\nw3af found too many consecutive errors while performing HTTP requests. In most cases this means that the remote web server is not reachable anymore, the network is down, or a WAF is blocking our tests. The last exception message was \"HTTP timeout error.\" (URLTimeoutError).\r\n\r\n## Scan details\r\n * Log: https://gist.github.com/andresriancho/5c5d5a3cbd324e1fe3d6\r\n * Pcap: https://www.dropbox.com/s/iotw36jyjmn6d12/w3af-failing-scan.pcap?dl=1\r\n\r\n## Specific timeout analysis\r\nIn the logs:\r\n```\r\n[jue 05 mar 2015 07:05:37 ART - debug] Re-sending request \"<HTTPRequest \"http://10.5.6.28/dvwa/vulnerabilities/fi/?page=htTps://w3af.org/\" (cookies:True, cache:True)>\" after initial exception: \"HTTP timeout error.\"\r\n[jue 05 mar 2015 07:05:37 ART - debug] Traceback for this error: Traceback (most recent call last):\r\n  File \"/home/pablo/pch/w3af/w3af/core/data/url/extended_urllib.py\", line 575, in _send\r\n    res = self._opener.open(req)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 404, in open\r\n    response = self._open(req, data)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 422, in _open\r\n    '_open', req)\r\n  File \"/usr/lib/python2.7/urllib2.py\", line 382, in _call_chain\r\n    result = func(*args)\r\n  File \"/home/pablo/pch/w3af/w3af/core/data/url/handlers/keepalive/__init__.py\", line 487, in http_open\r\n    return self.do_open(req)\r\n  File \"/home/pablo/pch/w3af/w3af/core/data/url/handlers/keepalive/__init__.py\", line 353, in do_open\r\n    raise URLTimeoutError()\r\nURLTimeoutError: HTTP timeout error.\r\n```\r\n\r\nIn the `pcap` file (note the time column, where we can clearly see that the remote end isn't answering the request and then we send the FIN packet) (tcp.stream eq 87):\r\n\r\n![close](https://cloud.githubusercontent.com/assets/865200/6503374/18c1f1a2-c30b-11e4-856d-c5bca8a66988.png)\r\n\r\nThe feature I added for preventing errors like this isn't working as expected, it escalates quickly from waiting 0.15 seconds to 1.5, which is not what I wanted\r\n\r\n```\r\n[pablo:/home/pablo/pch/w3af] develop 130 ± grep 'seconds before sending HTTP' /tmp/output-errors.txt \r\n[jue 05 mar 2015 07:05:37 ART - debug] Sleeping for 0.15 seconds before sending HTTP request after receiving URL/socket error. The total error count is at 1.\r\n[jue 05 mar 2015 07:05:38 ART - debug] Sleeping for 1.5 seconds before sending HTTP request after receiving URL/socket error. The total error count is at 10.\r\n[jue 05 mar 2015 07:05:39 ART - debug] Sleeping for 1.5 seconds before sending HTTP request after receiving URL/socket error. The total error count is at 10.\r\n[jue 05 mar 2015 07:05:40 ART - debug] Sleeping for 1.5 seconds before sending HTTP request after receiving URL/socket error. The total error count is at 10.\r\n[jue 05 mar 2015 07:05:42 ART - debug] Sleeping for 1.5 seconds before sending HTTP request after receiving URL/socket error. The total error count is at 10.\r\n[jue 05 mar 2015 07:05:43 ART - debug] Sleeping for 1.5 seconds before sending HTTP request after receiving URL/socket error. The total error count is at 10.\r\n[pablo:/home/pablo/pch/w3af] develop ± \r\n```\r\n\r\n## Profile\r\n```ini\r\n[profile]\r\ndescription = output_html_error\r\nname = test_error\r\n\r\n[grep.error_pages]\r\n\r\n[crawl.web_spider]\r\nonly_forward = True\r\nfollow_regex = .*dvwa.*\r\nignore_regex = .*logout.*|.*setup.*|.*security\\.php.*|^((?!dvwa).)*$\r\n\r\n[auth.detailed]\r\nusername = user\r\npassword = user\r\nusername_field = username\r\npassword_field = password\r\nauth_url = http://10.5.6.18/dvwa/login.php\r\ncheck_url = http://10.5.6.18/dvwa/index.php\r\ncheck_string = Welcome\r\ndata_format = %u=%U&%p=%P&Login=Login\r\nmethod = POST\r\n\r\n[output.html_file]\r\noutput_file = /tmp/w3af_output_html_utf8_error.html\r\nverbose = True\r\n\r\n[output.console]\r\nverbose = True\r\n\r\n[target]\r\ntarget = http://10.5.6.18/dvwa/\r\n\r\n[misc-settings]\r\nfuzz_cookies = False\r\nfuzz_form_files = True\r\nfuzz_url_filenames = False\r\nfuzz_url_parts = False\r\nfuzzed_files_extension = gif\r\nfuzzable_headers = \r\nform_fuzzing_mode = tmb\r\nstop_on_first_exception = False\r\nmax_discovery_time = 120\r\ninterface = ppp0\r\nlocal_ip_address = 10.5.6.13\r\nnon_targets = \r\nmsf_location = /opt/metasploit3/bin/\r\n\r\n[http-settings]\r\ntimeout = 15\r\nheaders_file = \r\nbasic_auth_user = \r\nbasic_auth_passwd = \r\nbasic_auth_domain = \r\nntlm_auth_domain = \r\nntlm_auth_user = \r\nntlm_auth_passwd = \r\nntlm_auth_url = \r\ncookie_jar_file = \r\nignore_session_cookies = False\r\nproxy_port = 8080\r\nproxy_address = \r\nuser_agent = w3af.org\r\nrand_user_agent = False\r\nmax_file_size = 400000\r\nmax_http_retries = 2\r\nmax_requests_per_second = 0\r\nalways_404 = \r\nnever_404 = \r\nstring_match_404 = \r\nurl_parameter = \r\n\r\n[audit.rfd]\r\n\r\n[grep.symfony]\r\noverride = False\r\n\r\n[grep.file_upload]\r\n\r\n[grep.wsdl_greper]\r\n\r\n[grep.cross_domain_js]\r\nsecure_js_file = %ROOT_PATH%/plugins/grep/cross_domain_js/secure-js-sources.txt\r\n\r\n[grep.http_auth_detect]\r\n\r\n[grep.svn_users]\r\n\r\n[grep.http_in_body]\r\n\r\n[grep.xss_protection_header]\r\n\r\n[grep.private_ip]\r\n\r\n[grep.motw]\r\n\r\n[grep.code_disclosure]\r\n\r\n[grep.form_cleartext_password]\r\n\r\n[grep.blank_body]\r\n\r\n[grep.path_disclosure]\r\n\r\n[grep.strange_http_codes]\r\n\r\n[grep.credit_cards]\r\n\r\n[grep.websockets_links]\r\n\r\n[grep.csp]\r\n\r\n[grep.dom_xss]\r\n\r\n[grep.form_autocomplete]\r\n\r\n[grep.clamav]\r\nclamd_socket = /var/run/clamav/clamd.ctl\r\n\r\n[grep.html_comments]\r\n\r\n[grep.click_jacking]\r\n\r\n[grep.strange_parameters]\r\n\r\n[grep.url_session]\r\n\r\n[grep.dot_net_event_validation]\r\n\r\n[grep.objects]\r\n\r\n[grep.error_500]\r\n\r\n[grep.meta_tags]\r\n\r\n[grep.password_profiling]\r\n\r\n[grep.directory_indexing]\r\n\r\n[grep.lang]\r\n\r\n[grep.get_emails]\r\nonly_target_domain = True\r\n\r\n[grep.hash_analysis]\r\n\r\n[grep.strange_reason]\r\n\r\n[grep.user_defined_regex]\r\nsingle_regex = \r\nregex_file_path = %ROOT_PATH%/plugins/grep/user_defined_regex/empty.txt\r\n\r\n[grep.cache_control]\r\n\r\n[grep.strange_headers]\r\n\r\n[grep.ssn]\r\n\r\n[grep.oracle]\r\n\r\n[grep.feeds]\r\n\r\n[grep.analyze_cookies]\r\n\r\n[audit.file_upload]\r\n\r\n[audit.eval]\r\n\r\n[audit.un_ssl]\r\n\r\n[audit.os_commanding]\r\n\r\n[audit.lfi]\r\n\r\n[audit.sqli]\r\n\r\n[audit.preg_replace]\r\n\r\n[audit.mx_injection]\r\n\r\n[audit.generic]\r\n\r\n[audit.format_string]\r\n\r\n[audit.shell_shock]\r\n\r\n[audit.memcachei]\r\n\r\n[audit.ldapi]\r\n\r\n[audit.buffer_overflow]\r\n\r\n[audit.redos]\r\n\r\n[audit.global_redirect]\r\n\r\n[audit.xpath]\r\n\r\n[audit.cors_origin]\r\n\r\n[audit.htaccess_methods]\r\n\r\n[audit.dav]\r\n\r\n[audit.ssi]\r\n\r\n[audit.csrf]\r\n\r\n[audit.xss]\r\n\r\n[audit.ssl_certificate]\r\n\r\n[audit.xst]\r\n\r\n[audit.blind_sqli]\r\n\r\n[audit.phishing_vector]\r\n\r\n[audit.response_splitting]\r\n\r\n[audit.rfi]\r\n\r\n[audit.frontpage]\r\n\r\n```"}}