{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/1251","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/1251/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1251/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/1251/events","html_url":"https://github.com/scrapy/scrapy/issues/1251","id":79024982,"number":1251,"title":"High CPU","user":{"login":"ErikvdVen","id":5047027,"avatar_url":"https://avatars.githubusercontent.com/u/5047027?v=3","gravatar_id":"","url":"https://api.github.com/users/ErikvdVen","html_url":"https://github.com/ErikvdVen","followers_url":"https://api.github.com/users/ErikvdVen/followers","following_url":"https://api.github.com/users/ErikvdVen/following{/other_user}","gists_url":"https://api.github.com/users/ErikvdVen/gists{/gist_id}","starred_url":"https://api.github.com/users/ErikvdVen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ErikvdVen/subscriptions","organizations_url":"https://api.github.com/users/ErikvdVen/orgs","repos_url":"https://api.github.com/users/ErikvdVen/repos","events_url":"https://api.github.com/users/ErikvdVen/events{/privacy}","received_events_url":"https://api.github.com/users/ErikvdVen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-05-21T14:39:54Z","updated_at":"2015-05-28T15:51:47Z","closed_at":"2015-05-28T15:51:47Z","body":"Scrapy uses a lot of CPU on our server, 40% is the minimum (when running 1 spider) and most of the time it goes up to 99% (running multiple spiders at the same time). We're using a DigitalOcean droplet with 1 CPU. \r\n\r\nIs this normal? I tried already several things:\r\n- add sleep(0.1) to Pipelines\r\n- disabling Pipelines\r\n- Using Scrapyd with job persistance.\r\n- set the amount of jobs per cpu with settings like below.\r\nscrapy.cfg:\r\n\r\n````\r\n[scrapyd]    \r\nmax_proc = 0    \r\nmax_proc_per_cpu = 2\r\nWhich didn't help either. These are my stats when running a spider:\r\n````\r\n\r\nNothing helps. These are my stats when I run one scraper:\r\n\r\n````\r\n  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND\r\n 3221 scrapy    20   0  836724  75568   8432 R 40.5  7.4   1:04.31 python\r\n 3231 scrapy    20   0  836724  75568   8432 S  1.0  7.4   0:01.40 python\r\n 3234 scrapy    20   0  836724  75568   8432 S  1.0  7.4   0:01.42 python\r\n 3235 scrapy    20   0  836724  75568   8432 S  1.0  7.4   0:01.44 python\r\n 3228 scrapy    20   0  836724  75568   8432 S  0.7  7.4   0:01.43 python\r\n````\r\n\r\nI found out that if I set the variable ```CONCURRENT_REQUESTS = 1``` the CPU stays below the 40% when running 1 spider, but it takes 3 times as long to finish. \r\n\r\nI also tried to turn off every extension, middleware and pipeline, but that didn't reduce the CPU load either. Any ideas?"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/106436004","html_url":"https://github.com/scrapy/scrapy/issues/1251#issuecomment-106436004","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1251","id":106436004,"user":{"login":"nramirezuy","id":1042865,"avatar_url":"https://avatars.githubusercontent.com/u/1042865?v=3","gravatar_id":"","url":"https://api.github.com/users/nramirezuy","html_url":"https://github.com/nramirezuy","followers_url":"https://api.github.com/users/nramirezuy/followers","following_url":"https://api.github.com/users/nramirezuy/following{/other_user}","gists_url":"https://api.github.com/users/nramirezuy/gists{/gist_id}","starred_url":"https://api.github.com/users/nramirezuy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nramirezuy/subscriptions","organizations_url":"https://api.github.com/users/nramirezuy/orgs","repos_url":"https://api.github.com/users/nramirezuy/repos","events_url":"https://api.github.com/users/nramirezuy/events{/privacy}","received_events_url":"https://api.github.com/users/nramirezuy/received_events","type":"User","site_admin":false},"created_at":"2015-05-28T15:51:47Z","updated_at":"2015-05-28T15:51:47Z","body":"@ErikvdVen yw, if you have anymore doubts you can reach us on [`scrapy-users`](https://groups.google.com/forum/#!forum/scrapy-users) or `#scrapy` IRC."}}