{"action":"opened","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/131","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/131/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/131/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/131/events","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/131","id":72485530,"number":131,"title":"Ignoring Max Depth","user":{"login":"BenChirlin","id":908329,"avatar_url":"https://avatars.githubusercontent.com/u/908329?v=3","gravatar_id":"","url":"https://api.github.com/users/BenChirlin","html_url":"https://github.com/BenChirlin","followers_url":"https://api.github.com/users/BenChirlin/followers","following_url":"https://api.github.com/users/BenChirlin/following{/other_user}","gists_url":"https://api.github.com/users/BenChirlin/gists{/gist_id}","starred_url":"https://api.github.com/users/BenChirlin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BenChirlin/subscriptions","organizations_url":"https://api.github.com/users/BenChirlin/orgs","repos_url":"https://api.github.com/users/BenChirlin/repos","events_url":"https://api.github.com/users/BenChirlin/events{/privacy}","received_events_url":"https://api.github.com/users/BenChirlin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-05-01T18:04:28Z","updated_at":"2015-05-01T18:04:28Z","closed_at":null,"body":"Hi. I was wondering if there's a way to have a set max depth but then have that ignored for certain types of URLs. I saw the `fetchWhitelistedMimeTypesBelowMaxDepth` but the documentation seems to imply this is only applicable to assets and not actual URLs in the page.\r\n\r\nSpecifically I want to crawl my site from an origin request to a given depth but want to ignore that depth with pagination links on pages to ensure the spider will crawl an entire paginated post and not cut off after a certain number of pages due to `maxDepth`. \r\n\r\nFor instance, if `spider.maxDepth = 3;` and the spider is on a page `www.mysite.com/2014/06/astory.html`, a depth 3 from it's origin request. Is there a way to ensure the spider will continue to crawl a link `www.mysite.com/2014/06/astory.html?pg=2` even though it's already at its `maxDepth`?\r\n\r\nThank you~"}}