{"action":"created","issue":{"url":"https://api.github.com/repos/PuerkitoBio/gocrawl/issues/44","labels_url":"https://api.github.com/repos/PuerkitoBio/gocrawl/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/PuerkitoBio/gocrawl/issues/44/comments","events_url":"https://api.github.com/repos/PuerkitoBio/gocrawl/issues/44/events","html_url":"https://github.com/PuerkitoBio/gocrawl/issues/44","id":109417863,"number":44,"title":"Crawl timeout","user":{"login":"juliend2","id":30395,"avatar_url":"https://avatars.githubusercontent.com/u/30395?v=3","gravatar_id":"","url":"https://api.github.com/users/juliend2","html_url":"https://github.com/juliend2","followers_url":"https://api.github.com/users/juliend2/followers","following_url":"https://api.github.com/users/juliend2/following{/other_user}","gists_url":"https://api.github.com/users/juliend2/gists{/gist_id}","starred_url":"https://api.github.com/users/juliend2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juliend2/subscriptions","organizations_url":"https://api.github.com/users/juliend2/orgs","repos_url":"https://api.github.com/users/juliend2/repos","events_url":"https://api.github.com/users/juliend2/events{/privacy}","received_events_url":"https://api.github.com/users/juliend2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-02T02:03:21Z","updated_at":"2015-10-02T16:04:41Z","closed_at":null,"body":"I would like the crawls to stop after 1 hour, because some websites have an infinite amount of URLs.\r\nSo I think it would be great to have a `CrawlTimeout` option, or an Extender function like `ShouldFinishCrawl() bool`.\r\n\r\nNote: I use it with SameHostOnly = true, to crawl only one site."},"comment":{"url":"https://api.github.com/repos/PuerkitoBio/gocrawl/issues/comments/145072312","html_url":"https://github.com/PuerkitoBio/gocrawl/issues/44#issuecomment-145072312","issue_url":"https://api.github.com/repos/PuerkitoBio/gocrawl/issues/44","id":145072312,"user":{"login":"PuerkitoBio","id":692384,"avatar_url":"https://avatars.githubusercontent.com/u/692384?v=3","gravatar_id":"","url":"https://api.github.com/users/PuerkitoBio","html_url":"https://github.com/PuerkitoBio","followers_url":"https://api.github.com/users/PuerkitoBio/followers","following_url":"https://api.github.com/users/PuerkitoBio/following{/other_user}","gists_url":"https://api.github.com/users/PuerkitoBio/gists{/gist_id}","starred_url":"https://api.github.com/users/PuerkitoBio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PuerkitoBio/subscriptions","organizations_url":"https://api.github.com/users/PuerkitoBio/orgs","repos_url":"https://api.github.com/users/PuerkitoBio/repos","events_url":"https://api.github.com/users/PuerkitoBio/events{/privacy}","received_events_url":"https://api.github.com/users/PuerkitoBio/received_events","type":"User","site_admin":false},"created_at":"2015-10-02T16:04:41Z","updated_at":"2015-10-02T16:04:41Z","body":"Hi Julien,\r\n\r\nYou can set the `MaxVisits` option if you know how many you want to visit, and of course you can stop returning URLs to visit in your `Visit` method, so you stop adding new links at some point.\r\n\r\nI believe you can also call Crawler.Stop() after some time? I'm not sure how well that works to be honest, haven't used gocrawl in a long time.\r\n\r\nHTH,\r\nMartin"}}