{"action":"created","issue":{"url":"https://api.github.com/repos/CrawlScript/WebCollector/issues/15","labels_url":"https://api.github.com/repos/CrawlScript/WebCollector/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/CrawlScript/WebCollector/issues/15/comments","events_url":"https://api.github.com/repos/CrawlScript/WebCollector/issues/15/events","html_url":"https://github.com/CrawlScript/WebCollector/issues/15","id":114782978,"number":15,"title":"没有爬取地址完成就退出","user":{"login":"qiangmzsx","id":4846144,"avatar_url":"https://avatars.githubusercontent.com/u/4846144?v=3","gravatar_id":"","url":"https://api.github.com/users/qiangmzsx","html_url":"https://github.com/qiangmzsx","followers_url":"https://api.github.com/users/qiangmzsx/followers","following_url":"https://api.github.com/users/qiangmzsx/following{/other_user}","gists_url":"https://api.github.com/users/qiangmzsx/gists{/gist_id}","starred_url":"https://api.github.com/users/qiangmzsx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qiangmzsx/subscriptions","organizations_url":"https://api.github.com/users/qiangmzsx/orgs","repos_url":"https://api.github.com/users/qiangmzsx/repos","events_url":"https://api.github.com/users/qiangmzsx/events{/privacy}","received_events_url":"https://api.github.com/users/qiangmzsx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-03T10:18:36Z","updated_at":"2015-11-15T06:53:33Z","closed_at":null,"body":"long start = System.currentTimeMillis();\r\n\t\tTonghuashunWebCollector crawler = new TonghuashunWebCollector(\"D:/var\");\r\n\t\tMap<String, BasicInfo> biMap = BasicInfoCache.getMarketStockMap();\r\n\t\tint count =0;\r\n\t\t\r\n\t\tfor (Map.Entry<String, BasicInfo> bi : biMap.entrySet()) {\r\n            // 接口中以前缀识别交易所\r\n            String prefix = \"\";\r\n            if (bi.getValue().getExchange_name().length()>1) {\r\n            \tprefix = bi.getValue().getExchange_name()+\"_\" + bi.getValue().getClass_code();\r\n            }\r\n            StringBuffer sb = new StringBuffer();\r\n            sb.append(prefix);\r\n            // 收集满MAX_NUM后，形成一个url，push进去\r\n            String url = PREFIX_URL_ + sb.toString()+ SUFFIX_URL_ +\"?\"+sb.toString();\r\n            crawler.addSeed(url);\r\n            logger.info(url);\r\n            count++;\r\n\t\t\tif (count == 10) {\r\n\t\t\t\tbreak;\r\n\t\t\t}\r\n        }\r\n\t\tcrawler.setThreads(5);\r\n\t\tcrawler.setMaxRetry(2);\r\n\t\tcrawler.start(1);\r\n\t\t\r\n\t\tSystem.out.println(System.currentTimeMillis()-start);\r\n\r\n代码如上，但是设置了但是仅仅爬了几个就退出了"},"comment":{"url":"https://api.github.com/repos/CrawlScript/WebCollector/issues/comments/156786735","html_url":"https://github.com/CrawlScript/WebCollector/issues/15#issuecomment-156786735","issue_url":"https://api.github.com/repos/CrawlScript/WebCollector/issues/15","id":156786735,"user":{"login":"hujunxianligong","id":3381345,"avatar_url":"https://avatars.githubusercontent.com/u/3381345?v=3","gravatar_id":"","url":"https://api.github.com/users/hujunxianligong","html_url":"https://github.com/hujunxianligong","followers_url":"https://api.github.com/users/hujunxianligong/followers","following_url":"https://api.github.com/users/hujunxianligong/following{/other_user}","gists_url":"https://api.github.com/users/hujunxianligong/gists{/gist_id}","starred_url":"https://api.github.com/users/hujunxianligong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hujunxianligong/subscriptions","organizations_url":"https://api.github.com/users/hujunxianligong/orgs","repos_url":"https://api.github.com/users/hujunxianligong/repos","events_url":"https://api.github.com/users/hujunxianligong/events{/privacy}","received_events_url":"https://api.github.com/users/hujunxianligong/received_events","type":"User","site_admin":false},"created_at":"2015-11-15T06:53:33Z","updated_at":"2015-11-15T06:53:33Z","body":"将start()中的数字加大。在每轮爬取中，如果失败，默认重试三次，如果三次都失败了，放到下一次。"}}