{"action":"created","issue":{"url":"https://api.github.com/repos/binux/pyspider/issues/138","labels_url":"https://api.github.com/repos/binux/pyspider/issues/138/labels{/name}","comments_url":"https://api.github.com/repos/binux/pyspider/issues/138/comments","events_url":"https://api.github.com/repos/binux/pyspider/issues/138/events","html_url":"https://github.com/binux/pyspider/issues/138","id":58330149,"number":138,"title":"Crawl rate bottleneck","user":{"login":"laapsaap","id":2487212,"avatar_url":"https://avatars.githubusercontent.com/u/2487212?v=3","gravatar_id":"","url":"https://api.github.com/users/laapsaap","html_url":"https://github.com/laapsaap","followers_url":"https://api.github.com/users/laapsaap/followers","following_url":"https://api.github.com/users/laapsaap/following{/other_user}","gists_url":"https://api.github.com/users/laapsaap/gists{/gist_id}","starred_url":"https://api.github.com/users/laapsaap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laapsaap/subscriptions","organizations_url":"https://api.github.com/users/laapsaap/orgs","repos_url":"https://api.github.com/users/laapsaap/repos","events_url":"https://api.github.com/users/laapsaap/events{/privacy}","received_events_url":"https://api.github.com/users/laapsaap/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-20T09:55:13Z","updated_at":"2015-02-21T11:01:47Z","closed_at":null,"body":"I am trying to increase the crawl rate of pyspider, but I just cant get it above 80/sec.\r\n\r\nMy setup is\r\n1 server with schedule result_worker webui processor  (low CPU load, 70% memory usage)\r\n3 servers with fetcher  (low load)\r\n3x RabbitMQ servers in Cluster, loadbalanced using HAproxy   (low load)\r\n1 MySQL server (high spec)   (low CPU load, 70% memory usage)\r\n\r\n\r\nI checked all the servers, but I cant find the bottleneck. If I set the crawl and burst rate to 150/s, I dont see any improvements in crawl rate.\r\n\r\nAny idea how I can improve the crawl rate? 80/s is just too slow for the amount servers I have deployed and scaling the fetchers horizontally doesnt affect performance."},"comment":{"url":"https://api.github.com/repos/binux/pyspider/issues/comments/75366852","html_url":"https://github.com/binux/pyspider/issues/138#issuecomment-75366852","issue_url":"https://api.github.com/repos/binux/pyspider/issues/138","id":75366852,"user":{"login":"binux","id":646451,"avatar_url":"https://avatars.githubusercontent.com/u/646451?v=3","gravatar_id":"","url":"https://api.github.com/users/binux","html_url":"https://github.com/binux","followers_url":"https://api.github.com/users/binux/followers","following_url":"https://api.github.com/users/binux/following{/other_user}","gists_url":"https://api.github.com/users/binux/gists{/gist_id}","starred_url":"https://api.github.com/users/binux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binux/subscriptions","organizations_url":"https://api.github.com/users/binux/orgs","repos_url":"https://api.github.com/users/binux/repos","events_url":"https://api.github.com/users/binux/events{/privacy}","received_events_url":"https://api.github.com/users/binux/received_events","type":"User","site_admin":false},"created_at":"2015-02-21T11:01:47Z","updated_at":"2015-02-21T11:01:47Z","body":"What is the status of message queue(by checking the message number in each\nqueue via web interface of rabbitmq) when crawl rate become stable.\n"}}