{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/170","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/170/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/170/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/170/events","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/170","id":107160077,"number":170,"title":"Prevent timeout events from being emitted when item is already fetched","user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-18T08:58:22Z","updated_at":"2015-09-18T16:05:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/pulls/170","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/170","diff_url":"https://github.com/cgiffard/node-simplecrawler/pull/170.diff","patch_url":"https://github.com/cgiffard/node-simplecrawler/pull/170.patch"},"body":"Fixes #136. It was possible for timeout events to be emitted after a page had already emitted a 404 event. The reason for this is likely the fact that when 404's are handled, the HTTP response socket is closed, but the request socket isn't. I'm not too familiar with the node http api, but I'm guessing this is why the timeout event for that request can still be emitted.\r\n\r\nJust like @fantapop thought, this was likely introduced in 40a71c1c5abffc2729316f4a90a0cb70b056612f, and this change makes simplecrawler behave the way it did before that commit.\r\n\r\nPerhaps what's really needed here is for the request socket to be closed together with the response socket (once again, I haven't worked much with the node `http` module directly), but this fixes the problem until further notice anyway."},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/141494339","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/170#issuecomment-141494339","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/170","id":141494339,"user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"created_at":"2015-09-18T16:05:13Z","updated_at":"2015-09-18T16:05:13Z","body":"This was a difficult one to reproduce in a test actually. Using a local URL, I'm having a hard time getting it to fail properly.\r\n\r\nThe example below uses a  URL that triggers the problem I see if I test this in production, though. Can you make any sense of why this URL would trigger this behavior, but not a local `/410`?\r\n\r\nI've tried setting a `Connection: close` header on local responses, but that didn't help. The server in question responds in HTTP 1.0, but I don't know if that makes a difference or not.\r\n\r\n@cgiffard let me know if you can reproduce the problem using the provided test case.\r\n\r\n```js\r\nit(\"should not emit both 404 and timeout events\", function(done) {\r\n\r\n    this.timeout(\"7s\");\r\n\r\n    var localCrawler = new Crawler(\"www.jackan.com\", \"/sida/0%29;\");\r\n    var didTimeout = false;\r\n    localCrawler.timeout = 1000;\r\n\r\n    localCrawler.on(\"fetchtimeout\", function(queueItem) {\r\n        console.log(\"fetchtimeout\", queueItem.url);\r\n        didTimeout = true;\r\n    });\r\n\r\n    localCrawler.on(\"fetch404\", function() {\r\n        setTimeout(function() {\r\n            didTimeout.should.equal(false);\r\n            done();\r\n        }, localCrawler.timeout * 4);\r\n    });\r\n\r\n    localCrawler.start();\r\n});\r\n\r\n```"}}