{"action":"created","issue":{"url":"https://api.github.com/repos/binux/pyspider/issues/181","labels_url":"https://api.github.com/repos/binux/pyspider/issues/181/labels{/name}","comments_url":"https://api.github.com/repos/binux/pyspider/issues/181/comments","events_url":"https://api.github.com/repos/binux/pyspider/issues/181/events","html_url":"https://github.com/binux/pyspider/issues/181","id":68476925,"number":181,"title":"What is correct infrastructure for the use-case?","user":{"login":"virtuman","id":1087996,"avatar_url":"https://avatars.githubusercontent.com/u/1087996?v=3","gravatar_id":"","url":"https://api.github.com/users/virtuman","html_url":"https://github.com/virtuman","followers_url":"https://api.github.com/users/virtuman/followers","following_url":"https://api.github.com/users/virtuman/following{/other_user}","gists_url":"https://api.github.com/users/virtuman/gists{/gist_id}","starred_url":"https://api.github.com/users/virtuman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/virtuman/subscriptions","organizations_url":"https://api.github.com/users/virtuman/orgs","repos_url":"https://api.github.com/users/virtuman/repos","events_url":"https://api.github.com/users/virtuman/events{/privacy}","received_events_url":"https://api.github.com/users/virtuman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-04-14T19:34:11Z","updated_at":"2015-04-15T22:12:10Z","closed_at":null,"body":"We currently scrape approximately 20 sites. \r\n\r\nEach of these 10 sites has an average of 1000 to 10000 pages that we scrape, parse and store parsed results to elasticsearch with the TTL age of 10 to 30 minutes.\r\n\r\nWe already have problems and pyspider fails, but we would really like to be able to scrape 100-500 sites like that.\r\n\r\nWe would like to be able to scrape all these pages for changes as frequently as possible and discover+scrape+parse+store new pages as soon as they get published on those external websites.\r\n\r\n## HARDWARE SERVER SPECS\r\nWe tried setting this up with all processes except for actual elasticsearch cluster on a single server with the following specs:\r\n* Linux CentOS 6.x distro with all latest updates\r\n* Python 2.6 / 2.7.9 / 3.4  -- tried them all, have all 3 installed\r\n* 128GB RAM\r\n* Multiple CPUs Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz - 32 cores total\r\n* RAID1 SSD disk (local) + RAID1 scrape result storage disk (local)\r\n* 1 gigabit dedicated line with guaranteed bandwidth of at least 500mbp/s used specifically for scraping only\r\n\r\nRabbitMQ is running on the same server for queue management\r\n![screen shot 2015-04-14 at 12 40 42 pm](https://cloud.githubusercontent.com/assets/1087996/7145780/92a1f41a-e2a3-11e4-9885-0974cb249a52.png)\r\n\r\n## STARTUP COMMAND\r\nWe use the following command to fire up pyspider to start all processes (our production environment, please correct me if we're not doing it correctly):\r\n```\r\n/usr/local/bin/python2.7 /usr/local/bin/pyspider -c ./config.json --queue-maxsize 20000 all --processor-num 100 --fetcher-num 37 --result-worker-num 10 --run-in subprocess\r\n```\r\nall pyspider projects are executed @every 3 minutes\r\nage on documents 30 minutes\r\n\r\n## CURRENT CUSTOMIZATION\r\nWe didn't find any packaged proxy management tools that would allow us to rotate through proxies and and we added a couple of methods that basically load full list of proxies and rotate them on each request for each scraping project so that every request comes from a unique IP address and it won't be reused on the same project until it runs through list of all available proxies\r\n\r\n##PROBLEM\r\nBy having all of this setup as described above, it appears that we often overload QUEUE and many processes never end up getting picked up and by then second round of crawlers will have re-scraped pages and re-attempt new parsing.\r\n\r\n## QUESTIONS\r\n1. The startup command that we use - is it correct to use it this way?\r\n2. Is RabbitMQ resource exhaustive? Web UI console is showing practically no load, and my assumption is that pyspider spends more time on processing / parsing than crawling. Should we move RabbitMQ to separate server/cluster? Would that result in noticeable improvements?\r\n3. Can scheduler be moved out to a separate server? Is it resource intensive on its own or is it low-resource-utilization part of pyspider?\r\n4. Can or should we move any other parts / modules of pySpider to separate servers to see significant improvements?\r\n5. Any other suggestions / recommendations that we should consider in order to optimize and improve scrape/parse speed so that it at least finishes tasks sometimes?\r\n6. Does pySpider have an ability to auto-re-scrape any expired page so that we don't need to re-add it to the queue manually? (currently all our projects originate on home page and will run through all available links, but if the link was removed from the website - we don't have an easy way to track what happened to that page, whether the product that was associated with that page is discontinued/no longer available or not since we no longer get updates on the page that is no longer found via pages that we scrape in the init scripts)?\r\n\r\n\r\n\r\nThank you very much for all your help and suggestions"},"comment":{"url":"https://api.github.com/repos/binux/pyspider/issues/comments/93584074","html_url":"https://github.com/binux/pyspider/issues/181#issuecomment-93584074","issue_url":"https://api.github.com/repos/binux/pyspider/issues/181","id":93584074,"user":{"login":"laapsaap","id":2487212,"avatar_url":"https://avatars.githubusercontent.com/u/2487212?v=3","gravatar_id":"","url":"https://api.github.com/users/laapsaap","html_url":"https://github.com/laapsaap","followers_url":"https://api.github.com/users/laapsaap/followers","following_url":"https://api.github.com/users/laapsaap/following{/other_user}","gists_url":"https://api.github.com/users/laapsaap/gists{/gist_id}","starred_url":"https://api.github.com/users/laapsaap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laapsaap/subscriptions","organizations_url":"https://api.github.com/users/laapsaap/orgs","repos_url":"https://api.github.com/users/laapsaap/repos","events_url":"https://api.github.com/users/laapsaap/events{/privacy}","received_events_url":"https://api.github.com/users/laapsaap/received_events","type":"User","site_admin":false},"created_at":"2015-04-15T22:12:10Z","updated_at":"2015-04-15T22:12:10Z","body":"I use pyspider to crawl all the pages and do a very simple regex to filter out the information i need and store it in a mysql database. And then I have another script that process the data from resultsdb and saving it in our own tables.\r\n\r\nI am pretty sure there are no performance benefits saving pages to disk vs database when you are doing a lot of pages (inode limits). After you process your crawled pages you still need to save the result somewhere right? \r\n\r\nI use regex mainly but in some of my scripts also beautiful soup, depends on the site.\r\n\r\nWhat proxies are you using?  I proxy all my outgoing requests by using a wrapper on fetcher."}}