{"action":"created","issue":{"url":"https://api.github.com/repos/binux/pyspider/issues/125","labels_url":"https://api.github.com/repos/binux/pyspider/issues/125/labels{/name}","comments_url":"https://api.github.com/repos/binux/pyspider/issues/125/comments","events_url":"https://api.github.com/repos/binux/pyspider/issues/125/events","html_url":"https://github.com/binux/pyspider/issues/125","id":57055840,"number":125,"title":"pyspider not crawling anymore after taskdb has grown","user":{"login":"laapsaap","id":2487212,"avatar_url":"https://avatars.githubusercontent.com/u/2487212?v=3","gravatar_id":"","url":"https://api.github.com/users/laapsaap","html_url":"https://github.com/laapsaap","followers_url":"https://api.github.com/users/laapsaap/followers","following_url":"https://api.github.com/users/laapsaap/following{/other_user}","gists_url":"https://api.github.com/users/laapsaap/gists{/gist_id}","starred_url":"https://api.github.com/users/laapsaap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laapsaap/subscriptions","organizations_url":"https://api.github.com/users/laapsaap/orgs","repos_url":"https://api.github.com/users/laapsaap/repos","events_url":"https://api.github.com/users/laapsaap/events{/privacy}","received_events_url":"https://api.github.com/users/laapsaap/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-09T16:43:48Z","updated_at":"2015-02-11T13:19:19Z","closed_at":null,"body":"I am trying to find out why pyspider is not crawling anymore. The task db has 30 million records, 17mil pending and 13mil success.\r\n\r\nAfter restarting all the pyspider components, it does a \"SELECT `taskid`,`project`,`schedule` FROM `site1` WHERE `status` = 1\" and then nothing happens. No crawl, everything is idle, no mysql queries and server load is 0.\r\n\r\nI cant really see what is wrong, right now I assume the taskdb is too big for pyspider to handle. I even tried running it on a 32gb memory server.\r\n\r\nDo you have any idea what can go wrong in this scenario?  Thanks"},"comment":{"url":"https://api.github.com/repos/binux/pyspider/issues/comments/73879320","html_url":"https://github.com/binux/pyspider/issues/125#issuecomment-73879320","issue_url":"https://api.github.com/repos/binux/pyspider/issues/125","id":73879320,"user":{"login":"binux","id":646451,"avatar_url":"https://avatars.githubusercontent.com/u/646451?v=3","gravatar_id":"","url":"https://api.github.com/users/binux","html_url":"https://github.com/binux","followers_url":"https://api.github.com/users/binux/followers","following_url":"https://api.github.com/users/binux/following{/other_user}","gists_url":"https://api.github.com/users/binux/gists{/gist_id}","starred_url":"https://api.github.com/users/binux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binux/subscriptions","organizations_url":"https://api.github.com/users/binux/orgs","repos_url":"https://api.github.com/users/binux/repos","events_url":"https://api.github.com/users/binux/events{/privacy}","received_events_url":"https://api.github.com/users/binux/received_events","type":"User","site_admin":false},"created_at":"2015-02-11T13:19:19Z","updated_at":"2015-02-11T13:19:19Z","body":"\"a timeout value\" is a timeout for `Scheduler._load_tasks`? It may not a reliable solution as it depends on the performance of database.\r\n\r\nTo build a large capacity scheduler, memory is the limitation. There are two aspect for the limitation. First, users always prefer lower resource consumption. Second, the design of scheduler, tasks of one project must be stored in memory of the same host. That's say, even if you can distribute different project to different hosts, the tasks of the same project is limited by the memory capacity of a single server (about hundred millions). Only if design a scheduler that can distribute the tasks to different instance (which is too complex for pyspider).\r\n\r\nAnother limitation is the performance of database. Reflected in the time of loading tasks into memory and the performance of query and update new requests.\r\n\r\nCurrently solution is to use `priority` and handwritten script to reduce the tasks in ACTIVE status and the number of new requests. With distributed mongodb cluster, one of the project in a pyspider instance has 300 million tasks in it (but not more than 1 hundred thousand tasks in ACTIVE).\r\n\r\nBut if you do want to have that large number of ACTIVE tasks, not loading them into memory in one-time is a reasonable solution as the tasks couldn't be fetched in one day. It's used in the spider of Baidu too. We select the tasks in order of priority with the number that can be done in 24 hours every day. You can design such mechanism too."}}