{"action":"created","issue":{"url":"https://api.github.com/repos/yasserg/crawler4j/issues/40","labels_url":"https://api.github.com/repos/yasserg/crawler4j/issues/40/labels{/name}","comments_url":"https://api.github.com/repos/yasserg/crawler4j/issues/40/comments","events_url":"https://api.github.com/repos/yasserg/crawler4j/issues/40/events","html_url":"https://github.com/yasserg/crawler4j/issues/40","id":63598793,"number":40,"title":"amazon api spinder","user":{"login":"frankskywalker","id":3746231,"avatar_url":"https://avatars.githubusercontent.com/u/3746231?v=3","gravatar_id":"","url":"https://api.github.com/users/frankskywalker","html_url":"https://github.com/frankskywalker","followers_url":"https://api.github.com/users/frankskywalker/followers","following_url":"https://api.github.com/users/frankskywalker/following{/other_user}","gists_url":"https://api.github.com/users/frankskywalker/gists{/gist_id}","starred_url":"https://api.github.com/users/frankskywalker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frankskywalker/subscriptions","organizations_url":"https://api.github.com/users/frankskywalker/orgs","repos_url":"https://api.github.com/users/frankskywalker/repos","events_url":"https://api.github.com/users/frankskywalker/events{/privacy}","received_events_url":"https://api.github.com/users/frankskywalker/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-23T01:29:01Z","updated_at":"2015-03-28T06:19:04Z","closed_at":null,"body":"hi yasserg\r\n\r\ncrawler4j helps me to get data from amazon api, there are many many urls to  query, first i add 2*numberOfCrawlers seeds and call Controller.startNonBlocking(), then i add another url when MyCrawler.visit() is called.But i find everytime the crawler4j will stop at a random task.Maybe 180 ,maybe 1000, there still other seeds but it seems the threads are all died.\r\n\r\nSo,what happened?can you help me?"},"comment":{"url":"https://api.github.com/repos/yasserg/crawler4j/issues/comments/87174725","html_url":"https://github.com/yasserg/crawler4j/issues/40#issuecomment-87174725","issue_url":"https://api.github.com/repos/yasserg/crawler4j/issues/40","id":87174725,"user":{"login":"laxika","id":1862574,"avatar_url":"https://avatars.githubusercontent.com/u/1862574?v=3","gravatar_id":"","url":"https://api.github.com/users/laxika","html_url":"https://github.com/laxika","followers_url":"https://api.github.com/users/laxika/followers","following_url":"https://api.github.com/users/laxika/following{/other_user}","gists_url":"https://api.github.com/users/laxika/gists{/gist_id}","starred_url":"https://api.github.com/users/laxika/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laxika/subscriptions","organizations_url":"https://api.github.com/users/laxika/orgs","repos_url":"https://api.github.com/users/laxika/repos","events_url":"https://api.github.com/users/laxika/events{/privacy}","received_events_url":"https://api.github.com/users/laxika/received_events","type":"User","site_admin":false},"created_at":"2015-03-28T06:19:04Z","updated_at":"2015-03-28T06:19:04Z","body":"Doesn't amazon banned you? I had to work on an Amazon crawling project (not with crawl4j) 2-3 weeks ago on my last workplace. You should query the API with 1 seconds delays otherwise it will ban you. Query amazon.com directly is even a worse business. We used 50+ proxies and 30 sec delays to finally be able and crawll all stuff. It took a toooong time but without the delay the whole proxy list is got banned after 30-40 mins and had to wait 2 days for unabnning.\r\n\r\nhttps://affiliate-program.amazon.com/gp/advertising/api/detail/faq.html"}}