{"action":"opened","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/1074","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/1074/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1074/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/1074/events","html_url":"https://github.com/scrapy/scrapy/issues/1074","id":62035528,"number":1074,"title":"0 delay inside utils/defer.py->defer_succeed makes reactor not to be async enough.","user":{"login":"aliowka","id":1184726,"avatar_url":"https://avatars.githubusercontent.com/u/1184726?v=3","gravatar_id":"","url":"https://api.github.com/users/aliowka","html_url":"https://github.com/aliowka","followers_url":"https://api.github.com/users/aliowka/followers","following_url":"https://api.github.com/users/aliowka/following{/other_user}","gists_url":"https://api.github.com/users/aliowka/gists{/gist_id}","starred_url":"https://api.github.com/users/aliowka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aliowka/subscriptions","organizations_url":"https://api.github.com/users/aliowka/orgs","repos_url":"https://api.github.com/users/aliowka/repos","events_url":"https://api.github.com/users/aliowka/events{/privacy}","received_events_url":"https://api.github.com/users/aliowka/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-03-16T10:20:16Z","updated_at":"2015-03-16T10:20:16Z","closed_at":null,"body":"The problem appears when HTTP_CACHE is enabled\r\nSuppose I need to collect items while using httpcache with FIleSystemStorage. \r\nIn the items pipeline I want to send them out with some twisted based client.\r\nWhile having HTTP_CACHE enabled I observe that the items are sent very slow and the memory is growing very fast. \r\nFor instance:\r\n1.When cache is disabled: I have about 50 items/second and the total process memory is 70M. 2.When it's enabled I have 3 items/second and 4G of memory used (my items are big). Fortunately my task is small (ends after 3 min.) \r\nWhen the crawling ends and spider closes, I observe the bulk of items being transfered by the client with 2K items/s speed.\r\nIt's clear for me that when the cache is enabled the reactor is not released for enough time to accomplish it's async tasks (items sending) and that what cause this abnormal behavior, that makes the using of cache very questionable. \r\nI found that there is a delay 0 in the   scrapy/scrapy/utils/defer.py : defer_succeed function:\r\nreactor.callLater(0, d.callback, result) \r\nthat seems just not enough for the cached responses that does not obey to DOWNLOAD_DELAY and potentially may through big bulks of items very fast (it does)\r\nI tried to increase the delay to 0.1 and this fixed the issue."}}