{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/991","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/991/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/991/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/991/events","html_url":"https://github.com/scrapy/scrapy/pull/991","id":52731850,"number":991,"title":"dupefilter should record requests even with dont_filter","user":{"login":"adon-at-work","id":8486533,"avatar_url":"https://avatars.githubusercontent.com/u/8486533?v=3","gravatar_id":"","url":"https://api.github.com/users/adon-at-work","html_url":"https://github.com/adon-at-work","followers_url":"https://api.github.com/users/adon-at-work/followers","following_url":"https://api.github.com/users/adon-at-work/following{/other_user}","gists_url":"https://api.github.com/users/adon-at-work/gists{/gist_id}","starred_url":"https://api.github.com/users/adon-at-work/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adon-at-work/subscriptions","organizations_url":"https://api.github.com/users/adon-at-work/orgs","repos_url":"https://api.github.com/users/adon-at-work/repos","events_url":"https://api.github.com/users/adon-at-work/events{/privacy}","received_events_url":"https://api.github.com/users/adon-at-work/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-12-23T11:04:27Z","updated_at":"2015-01-06T01:30:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/991","html_url":"https://github.com/scrapy/scrapy/pull/991","diff_url":"https://github.com/scrapy/scrapy/pull/991.diff","patch_url":"https://github.com/scrapy/scrapy/pull/991.patch"},"body":"dupefilter must see and record all requests even with dont_filter, and\r\nthus a subsequent duplicate request happened to have no dont_filter can\r\nbe filtered out.\r\n\r\nExample: consider having some start_urls marked as dont_filter, any\r\nself-referencing urls will not be de-duplicated.\r\n\r\nApparently this problem was introduced since Jun 2011 after a refactoring at\r\nhttps://github.com/scrapy/scrapy/commit/03751749a80526f3d97f60f0f63892501ef5de19"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/68813037","html_url":"https://github.com/scrapy/scrapy/pull/991#issuecomment-68813037","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/991","id":68813037,"user":{"login":"Curita","id":213781,"avatar_url":"https://avatars.githubusercontent.com/u/213781?v=3","gravatar_id":"","url":"https://api.github.com/users/Curita","html_url":"https://github.com/Curita","followers_url":"https://api.github.com/users/Curita/followers","following_url":"https://api.github.com/users/Curita/following{/other_user}","gists_url":"https://api.github.com/users/Curita/gists{/gist_id}","starred_url":"https://api.github.com/users/Curita/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Curita/subscriptions","organizations_url":"https://api.github.com/users/Curita/orgs","repos_url":"https://api.github.com/users/Curita/repos","events_url":"https://api.github.com/users/Curita/events{/privacy}","received_events_url":"https://api.github.com/users/Curita/received_events","type":"User","site_admin":false},"created_at":"2015-01-06T01:30:09Z","updated_at":"2015-01-06T01:30:09Z","body":"This behavior has been around for too long to openly break backward compatibility like that. Also this functionality is documented as \"ignoring the duplicates filter\" entirely (see [here](http://scrapy.readthedocs.org/en/latest/topics/request-response.html#request-objects) on `dont_filter` parameter), and it's shared with the OffsiteMiddleware (see [here](http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html#module-scrapy.contrib.spidermiddleware.offsite)), so it should be excepted.\r\n\r\nNonetheless, this could be a useful feature provided with a different flag like `dont_store` (which should be False to maintain how it's working right now). I'll suggest you open up a different pull request for that enhancement, with a test (it's too easy for interchange those two conditions without knowing) and documentation for the new parameter."}}