{"action":"created","issue":{"url":"https://api.github.com/repos/binux/pyspider/issues/130","labels_url":"https://api.github.com/repos/binux/pyspider/issues/130/labels{/name}","comments_url":"https://api.github.com/repos/binux/pyspider/issues/130/comments","events_url":"https://api.github.com/repos/binux/pyspider/issues/130/events","html_url":"https://github.com/binux/pyspider/issues/130","id":57573046,"number":130,"title":"Filter_urls","user":{"login":"vortex14","id":9502625,"avatar_url":"https://avatars.githubusercontent.com/u/9502625?v=3","gravatar_id":"","url":"https://api.github.com/users/vortex14","html_url":"https://github.com/vortex14","followers_url":"https://api.github.com/users/vortex14/followers","following_url":"https://api.github.com/users/vortex14/following{/other_user}","gists_url":"https://api.github.com/users/vortex14/gists{/gist_id}","starred_url":"https://api.github.com/users/vortex14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vortex14/subscriptions","organizations_url":"https://api.github.com/users/vortex14/orgs","repos_url":"https://api.github.com/users/vortex14/repos","events_url":"https://api.github.com/users/vortex14/events{/privacy}","received_events_url":"https://api.github.com/users/vortex14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-13T08:43:20Z","updated_at":"2015-02-13T08:51:32Z","closed_at":null,"body":"How do I get access to the array urls in the function detail_page ?\r\n\r\nclass Handler(BaseHandler):\r\n    #@config(age=0)\r\n    crawl_config = {\r\n        \"age\":0\r\n    }\r\n    \r\n    urls = []\r\n    start_url = 'http://gilt.com'\r\n    domain = 'gilt.com'          \r\n            \r\n     \r\n    \r\n    #@every(minutes=60 * 60)\r\n    \r\n    def on_start(self):\r\n        self.get_page(  self.start_url  )\r\n    \r\n    \r\n    def get_page(self,url):    \r\n        \r\n        self.urls.append(url)\r\n                 \r\n        self.crawl(url=url, callback=self.detail_page, timeout=10)\r\n        \r\n             \r\n                    \r\n               \r\n    @catch_status_code_error\r\n    def detail_page(self, response):\r\n        print (self.urls)\r\n        if response.status_code == 200:\r\n            if self.domain in response.url:\r\n                url = response.url\r\n                body = response.text\r\n            \r\n            \r\n                      \r\n                for each in response.doc('a[href^=\"http\"]').items():\r\n                \r\n                    if not (each.attr.href in self.urls):\r\n                        self.get_page(each.attr.href)"},"comment":{"url":"https://api.github.com/repos/binux/pyspider/issues/comments/74222617","html_url":"https://github.com/binux/pyspider/issues/130#issuecomment-74222617","issue_url":"https://api.github.com/repos/binux/pyspider/issues/130","id":74222617,"user":{"login":"binux","id":646451,"avatar_url":"https://avatars.githubusercontent.com/u/646451?v=3","gravatar_id":"","url":"https://api.github.com/users/binux","html_url":"https://github.com/binux","followers_url":"https://api.github.com/users/binux/followers","following_url":"https://api.github.com/users/binux/following{/other_user}","gists_url":"https://api.github.com/users/binux/gists{/gist_id}","starred_url":"https://api.github.com/users/binux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binux/subscriptions","organizations_url":"https://api.github.com/users/binux/orgs","repos_url":"https://api.github.com/users/binux/repos","events_url":"https://api.github.com/users/binux/events{/privacy}","received_events_url":"https://api.github.com/users/binux/received_events","type":"User","site_admin":false},"created_at":"2015-02-13T08:51:32Z","updated_at":"2015-02-13T08:51:32Z","body":"Don't do it like that, pyspider is not scrapy, you don't have to manage and deduplicate the urls yourself. Just submit the urls via `self.crawl` and pyspider will take care the deduplicate things for you.\r\n\r\n```\r\nclass Handler(BaseHandler):\r\n    def on_start(self):\r\n        self.crawl(start_url, callback=self.detail_page)\r\n\r\n    def detail_page(self, response):\r\n        for each in response.doc('a[href^=\"http\"]').items():\r\n            self.crawl(each.attr.href, callback=self.detail_page)\r\n        return {\r\n            ... # extract information here\r\n        }\r\n```\r\n\r\nAnd don't use class member variables, pyspider is designed as distributed, member variables may not be visitable in other instance."}}