{"action":"created","issue":{"url":"https://api.github.com/repos/scrapinghub/distributed-frontera/issues/6","labels_url":"https://api.github.com/repos/scrapinghub/distributed-frontera/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/scrapinghub/distributed-frontera/issues/6/comments","events_url":"https://api.github.com/repos/scrapinghub/distributed-frontera/issues/6/events","html_url":"https://github.com/scrapinghub/distributed-frontera/issues/6","id":106807650,"number":6,"title":"Why is \"TypeError\" being raised? - Unable to debug the cause","user":{"login":"stejesh","id":13462801,"avatar_url":"https://avatars.githubusercontent.com/u/13462801?v=3","gravatar_id":"","url":"https://api.github.com/users/stejesh","html_url":"https://github.com/stejesh","followers_url":"https://api.github.com/users/stejesh/followers","following_url":"https://api.github.com/users/stejesh/following{/other_user}","gists_url":"https://api.github.com/users/stejesh/gists{/gist_id}","starred_url":"https://api.github.com/users/stejesh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stejesh/subscriptions","organizations_url":"https://api.github.com/users/stejesh/orgs","repos_url":"https://api.github.com/users/stejesh/repos","events_url":"https://api.github.com/users/stejesh/events{/privacy}","received_events_url":"https://api.github.com/users/stejesh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-09-16T16:13:45Z","updated_at":"2015-09-22T07:33:52Z","closed_at":null,"body":"I've installed kafka, zookeeperd is running fine.\r\n\r\nand then I created the kafka topics as per the docs\r\n```sh\r\nkafka-topics.sh --create --topic frontier-todo --replication-factor 1 --partitions 2 --zookeeper localhost:2181\r\n$ kafka-topics.sh --create --topic frontier-done --replication-factor 1 --partitions 2 --zookeeper localhost:2181\r\n$ kafka-topics.sh --create --topic frontier-score --replication-factor 1 --partitions 1 --zookeeper localhost:2181\r\n```\r\n\r\nTo confirm if they've been created, I ran `kafkat partitions` and it shows the topics created\r\n\r\n```sh\r\nTopic\t\tPartition\tLeader\t\tReplicas\t\t\t\t\t\tISRs\t\t\t\t\r\nTutorialTopic\t0\t\t0\t\t[0]\t\t\t\t\t\t\t[0]\t\t\t\t\r\nfrontier-score\t0\t\t0\t\t[0]\t\t\t\t\t\t\t[0]\t\t\t\t\r\nfrontier-todo\t0\t\t0\t\t[0]\t\t\t\t\t\t\t[0]\t\t\t\t\r\nfrontier-todo\t1\t\t0\t\t[0]\t\t\t\t\t\t\t[0]\t\t\t\t\r\nfrontier-done\t0\t\t0\t\t[0]\t\t\t\t\t\t\t[0]\t\t\t\t\r\nfrontier-done\t1\t\t0\t\t[0]\t\t\t\t\t\t\t[0]\t\t\r\n```\t\t\r\n\r\n\r\n\r\n\r\nAfter that, while starting the DB Worker\r\n\r\non running\r\n\r\n` python -m distributed_frontera.worker.main --config crawler.settings`\r\n\r\nI get the following error\r\n\r\n```sh\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/runpy.py\", line 162, in _run_module_as_main\r\n    \"__main__\", fname, loader, pkg_name)\r\n  File \"/usr/lib/python2.7/runpy.py\", line 72, in _run_code\r\n    exec code in run_globals\r\n  File \"/home/tejesh/projects/billionleads/python/python2.7-env/lib/python2.7/site-packages/distributed_frontera/worker/main.py\", line 237, in <module>\r\n    worker = FrontierWorker(settings, args.no_batches, args.no_scoring, args.no_incoming)\r\n  File \"/home/tejesh/projects/billionleads/python/python2.7-env/lib/python2.7/site-packages/distributed_frontera/worker/main.py\", line 70, in __init__\r\n    max_buffer_size=10485760)\r\n  File \"/home/tejesh/projects/billionleads/python/python2.7-env/local/lib/python2.7/site-packages/kafka/consumer/simple.py\", line 126, in __init__\r\n    auto_commit_every_t=auto_commit_every_t)\r\n  File \"/home/tejesh/projects/billionleads/python/python2.7-env/local/lib/python2.7/site-packages/kafka/consumer/base.py\", line 49, in __init__\r\n    self.topic = kafka_bytestring(topic)\r\n  File \"/home/tejesh/projects/billionleads/python/python2.7-env/local/lib/python2.7/site-packages/kafka/util.py\", line 101, in kafka_bytestring\r\n    raise TypeError(s)\r\nTypeError: None\r\n```\r\n\r\nBasically `TypeError` is raised and i'm unable to debug this.\r\n\r\nWhat seems to be the issue?\r\n\r\nI've even tried testing kafka by sending messages\r\n`echo \"Hello, World\" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null`\r\n\r\nand it's working fine.\r\n\r\nHow should i proceed in this issue?\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scrapinghub/distributed-frontera/issues/comments/142202457","html_url":"https://github.com/scrapinghub/distributed-frontera/issues/6#issuecomment-142202457","issue_url":"https://api.github.com/repos/scrapinghub/distributed-frontera/issues/6","id":142202457,"user":{"login":"stejesh","id":13462801,"avatar_url":"https://avatars.githubusercontent.com/u/13462801?v=3","gravatar_id":"","url":"https://api.github.com/users/stejesh","html_url":"https://github.com/stejesh","followers_url":"https://api.github.com/users/stejesh/followers","following_url":"https://api.github.com/users/stejesh/following{/other_user}","gists_url":"https://api.github.com/users/stejesh/gists{/gist_id}","starred_url":"https://api.github.com/users/stejesh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stejesh/subscriptions","organizations_url":"https://api.github.com/users/stejesh/orgs","repos_url":"https://api.github.com/users/stejesh/repos","events_url":"https://api.github.com/users/stejesh/events{/privacy}","received_events_url":"https://api.github.com/users/stejesh/received_events","type":"User","site_admin":false},"created_at":"2015-09-22T07:33:52Z","updated_at":"2015-09-22T07:33:52Z","body":"@sibiryakov \r\n\r\nYes, I'm using the copied code from `distributed_frontera.worker.strategy.bfs` + implementing the `finsihed()` method\r\nas my `strategy0.py` file\r\n\r\n(added some `print` statements to debug)\r\n**Strategy0 Code:** http://pastebin.com/ynp1Ls9X \r\n\r\n\r\n\r\nI'm running it this way (both `--config` and `--strategy` point to same file)\r\n```sh\r\npython -m distributed_frontera.worker.score --config crawler.frontera.strategy_list.strategy0 --strategy crawler.frontera.strategy_list.strategy0\r\n```\r\n\r\ninstead of command specified in the docs below\r\n```sh\r\npython -m distributed_frontera.worker.score --config crawler.frontera.strategy_list.strategy0 --strategy distributed_frontera.worker.strategy.bfs\r\n```\r\n\r\nBecause i get a \"can't instantiate abstract class\" error when done by the docs\r\n```sh\r\n/distributed_frontera/worker/score.py\", line 39, in __init__\r\n    self.strategy = strategy_module.CrawlingStrategy()\r\nTypeError: Can't instantiate abstract class CrawlingStrategy with abstract methods finished\r\n```\r\n\r\nThe `finished()` method is abstract, and i cannot pass `distributed_frontera.worker.strategy.bfs` as --strategy\r\n\r\n**Full Spider Output here:** http://pastebin.com/SLtbvD5E\r\n\r\nI'm getting a lot of Timeout errors on distributed_frontera, this does not happen in frontera mode\r\n```sh\r\n[backend] Timeout (5.0 seconds) while trying to get 256 requests\r\n2015-09-22 12:18:31 [backend] WARNING: Timeout (5.0 seconds) while trying to get 256 requests\r\n```\r\n\r\n**Spider Code:** http://pastebin.com/8tC4YK1z\r\n\r\n**SW output:**  http://pastebin.com/L0QnYjgr\r\n\r\n**DB output:**  http://pastebin.com/J7ictTAE\r\n\r\n\r\n"}}